Processing queries based on level assignment information

ABSTRACT

A method for execution by a node includes determining first level assignment information indicating assignment to a first subset of levels of a plurality of levels and determining second level assignment information indicating assignment to a second subset of levels of the plurality of levels. The second level assignment information indicates a second compute sequence number that is different from a first compute sequence number indicated by the first level assignment information. A query indicating the first compute sequence number is determined for execution. Execution of at least one query operator of the query is performed in accordance with a level included in the first subset of levels based on determining to utilize the first level assignment information based on the query indicating the first compute sequence number.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present U.S. Utility Patent Application claims priority pursuant to35 U.S.C. § 120 as a continuation of U.S. Utility application Ser. No.17/340,553, entitled “LEVEL-BASED QUERIES IN A DATABASE SYSTEM ANDMETHODS FOR USE THEREWITH”, filed Jun. 7, 2021, which is acontinuation-in-part of U.S. Utility application Ser. No. 16/778,194,entitled “SERVICING CONCURRENT QUERIES VIA VIRTUAL SEGMENT RECOVERY”,filed Jan. 31, 2020, issued as U.S. Pat. No. 11,061,910 on Jul. 13,2021, both of which are hereby incorporated herein by reference in theirentirety and made part of the present U.S. Utility Patent Applicationfor all purposes.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable.

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not Applicable.

BACKGROUND OF THE INVENTION Technical Field of the Invention

This invention relates generally to computer networking and moreparticularly to database system and operation.

Description of Related Art

Computing devices are known to communicate data, process data, and/orstore data. Such computing devices range from wireless smart phones,laptops, tablets, personal computers (PC), work stations, and video gamedevices, to data centers that support millions of web searches, stocktrades, or on-line purchases every day. In general, a computing deviceincludes a central processing unit (CPU), a memory system, userinput/output interfaces, peripheral device interfaces, and aninterconnecting bus structure.

As is further known, a computer may effectively extend its CPU by using“cloud computing” to perform one or more computing functions (e.g., aservice, an application, an algorithm, an arithmetic logic function,etc.) on behalf of the computer. Further, for large services,applications, and/or functions, cloud computing may be performed bymultiple cloud computing resources in a distributed manner to improvethe response time for completion of the service, application, and/orfunction.

Of the many applications a computer can perform, a database system isone of the largest and most complex applications. In general, a databasesystem stores a large amount of data in a particular way for subsequentprocessing. In some situations, the hardware of the computer is alimiting factor regarding the speed at which a database system canprocess a particular function. In some other instances, the way in whichthe data is stored is a limiting factor regarding the speed ofexecution. In yet some other instances, restricted co-process optionsare a limiting factor regarding the speed of execution.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

FIG. 1 is a schematic block diagram of an embodiment of a large scaledata processing network that includes a database system in accordancewith the present invention;

FIG. 1A is a schematic block diagram of an embodiment of a databasesystem in accordance with the present invention;

FIG. 2 is a schematic block diagram of an embodiment of anadministrative sub-system in accordance with the present invention;

FIG. 3 is a schematic block diagram of an embodiment of a configurationsub-system in accordance with the present invention;

FIG. 4 is a schematic block diagram of an embodiment of a parallelizeddata input sub-system in accordance with the present invention;

FIG. 5 is a schematic block diagram of an embodiment of a parallelizedquery and response (Q&R) sub-system in accordance with the presentinvention;

FIG. 6 is a schematic block diagram of an embodiment of a parallelizeddata store, retrieve, and/or process (IO& P) sub-system in accordancewith the present invention;

FIG. 7 is a schematic block diagram of an embodiment of a computingdevice in accordance with the present invention;

FIG. 8 is a schematic block diagram of another embodiment of a computingdevice in accordance with the present invention;

FIG. 9 is a schematic block diagram of another embodiment of a computingdevice in accordance with the present invention;

FIG. 10 is a schematic block diagram of an embodiment of a node of acomputing device in accordance with the present invention;

FIG. 11 is a schematic block diagram of an embodiment of a node of acomputing device in accordance with the present invention;

FIG. 12 is a schematic block diagram of an embodiment of a node of acomputing device in accordance with the present invention;

FIG. 13 is a schematic block diagram of an embodiment of a node of acomputing device in accordance with the present invention;

FIG. 14 is a schematic block diagram of an embodiment of operatingsystems of a computing device in accordance with the present invention;

FIGS. 15-23 are schematic block diagrams of an example of processing atable or data set for storage in the database system in accordance withthe present invention;

FIGS. 24A-24F are schematic block diagrams of various embodiments of anode of a computing device that implements a segment scheduler module inaccordance with the present invention;

FIGS. 24G-24K are schematic block diagrams of an embodiment of a segmentscheduler module in accordance with the present invention;

FIGS. 24L-24M are logic diagrams illustrating a method of retrievingsegments for query execution based on drive utilization data inaccordance with the present invention;

FIGS. 25A-25B are schematic block diagrams of embodiments of a node thatimplements a segment recovery module to process virtual segments inaccordance with the present invention;

FIG. 25C illustrates an embodiment of a segment group in accordance withthe present invention;

FIGS. 25D-25F are schematic block diagrams of embodiments of a node thatimplements a segment recovery module to process virtual segments inaccordance with the present invention;

FIG. 25G illustrates an example embodiment of a segment that includes aplurality of data blocks;

FIGS. 25H-25I are schematic block diagrams of embodiments of a node thatimplements a segment recovery module to process virtual segments inaccordance with the present invention;

FIG. 25J is a logic diagram illustrating a method of utilizing arecovery scheme to process virtual segments in accordance with thepresent invention;

FIG. 25K is a logic diagram illustrating a method of utilizing a locallyrebuilt segment to service multiple concurrently executing queries inaccordance with the present invention;

FIG. 25L is a logic diagram illustrating a method of batching recoveryof multiple data portions in accordance with the present invention;

FIG. 26A is a schematic block diagrams of an embodiment of a node thatimplements a processing resource allocation module to allocateprocessing resources utilized to process segments in accordance with thepresent invention;

FIG. 26B-26C illustrate embodiment of a segment processing module thatutilizes a plurality of sets of parallel threads to concurrently processa plurality of segments in accordance with the present invention;

FIG. 26D is a logic diagram illustrating a method of allocating parallelthreads to process queries in accordance with the present invention;

FIG. 27A illustrates an embodiment of data ownership information inaccordance with the present invention;

FIG. 27B illustrates a timeline of a plurality of consensus protocolexecutions in accordance with the present invention;

FIG. 27C illustrates an example timeline of execution of a plurality ofqueries in accordance with the present invention;

FIG. 27D illustrates an example embodiment of multiple versions of dataownership information in accordance with the present invention;

FIGS. 27E and 27F are schematic block diagrams of an embodiment of twoexample node that implement a segment set generating module inaccordance with the present invention;

FIG. 27G illustrates an example timeline of execution of a plurality ofqueries in accordance with the present invention;

FIG. 27H illustrates an example embodiment of a version of dataownership information in accordance with the present invention;

FIGS. 27I and 27J are schematic block diagrams of an example embodimenta node that implement a segment set generating module in accordance withthe present invention;

FIG. 27K is a logic diagram illustrating a method of executing a querybased on a corresponding ownership sequence number in accordance withthe present invention;

FIG. 28A is a schematic block diagram of an embodiment of a queryexecution plan in accordance with the present invention;

FIG. 28B is a schematic block diagram of an embodiment of multiplecomputing clusters utilized to implement a query execution plan inaccordance with the present invention;

FIGS. 28C and 28D illustrate embodiments of level assignment informationin accordance with the present invention;

FIGS. 28E-28G are schematic block diagram of an example embodiment of acomputing cluster utilized to execute multiple queries in accordancewith the present invention;

FIG. 28H is a schematic block diagram of an embodiment of a node thatimplements a level participation determination module in accordance withthe present invention;

FIG. 28I illustrates a timeline of a plurality of consensus protocolexecutions in accordance with the present invention;

FIG. 28J illustrates an example timeline of execution of a plurality ofqueries in accordance with the present invention;

FIG. 28K illustrates an example embodiment of multiple versions of levelassignment information in accordance with the present invention;

FIG. 28L is a schematic block diagram of an example embodiment of a nodethat implements a level participation determination module in accordancewith the present invention;

FIG. 28M illustrates an example timeline of execution of a plurality ofqueries in accordance with the present invention;

FIG. 28N is a logic diagram illustrating a method of executing a querybased on selecting a parent node for processing a query resultant inaccordance with the present invention;

FIG. 28O is a logic diagram illustrating a method of executing a querybased on a corresponding compute sequence number in accordance with thepresent invention;

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 is a schematic block diagram of an embodiment of a large-scaledata processing network that includes data gathering devices (1, 1-1through 1-n), data systems (2, 2-1 through 2-N), data storage systems(3, 3-1 through 3-n), a network 4, and a database system 10. The datagathering devices are computing devices that collect a wide variety ofdata and may further include sensors, monitors, measuring instruments,and/or other instrument for collecting data. The data gathering devicescollect data in real-time (i.e., as it is happening) and provides it todata system 2-1 for storage and real-time processing of queries 5-1 toproduce responses 6-1. As an example, the data gathering devices arecomputing in a factory collecting data regarding manufacturing of one ormore products and the data system is evaluating queries to determinemanufacturing efficiency, quality control, and/or product developmentstatus.

The data storage systems 3 store existing data. The existing data mayoriginate from the data gathering devices or other sources, but the datais not real time data. For example, the data storage system storesfinancial data of a bank, a credit card company, or like financialinstitution. The data system 2-N processes queries 5-N regarding thedata stored in the data storage systems to produce responses 6-N.

Data system 2 processes queries regarding real time data from datagathering devices and/or queries regarding non-real time data stored inthe data storage system 3. The data system 2 produces responses inregard to the queries. Storage of real time and non-real time data, theprocessing of queries, and the generating of responses will be discussedwith reference to one or more of the subsequent figures.

FIG. 1A is a schematic block diagram of an embodiment of a databasesystem 10 that includes a parallelized data input sub-system 11, aparallelized data store, retrieve, and/or process sub-system 12, aparallelized query and response sub-system 13, system communicationresources 14, an administrative sub-system 15, and a configurationsub-system 16. The system communication resources 14 include one or moreof wide area network (WAN) connections, local area network (LAN)connections, wireless connections, wireline connections, etc. to couplethe sub-systems 11, 12, 13, 15, and 16 together.

Each of the sub-systems 11, 12, 13, 15, and 16 include a plurality ofcomputing devices; an example of which is discussed with reference toone or more of FIGS. 7-9 . Hereafter, the parallelized data inputsub-system 11 may also be referred to as a data input sub-system, theparallelized data store, retrieve, and/or process sub-system may also bereferred to as a data storage and processing sub-system, and theparallelized query and response sub-system 13 may also be referred to asa query and results sub-system.

In an example of operation, the parallelized data input sub-system 11receives a data set (e.g., a table) that includes a plurality ofrecords. A record includes a plurality of data fields. As a specificexample, the data set includes tables of data from a data source. Forexample, a data source includes one or more computers. As anotherexample, the data source is a plurality of machines. As yet anotherexample, the data source is a plurality of data mining algorithmsoperating on one or more computers.

As is further discussed with reference to FIG. 15 , the data sourceorganizes its records of the data set into a table that includes rowsand columns. The columns represent data fields of data for the rows.Each row corresponds to a record of data. For example, a table includepayroll information for a company's employees. Each row is an employee'spayroll record. The columns include data fields for employee name,address, department, annual salary, tax deduction information, directdeposit information, etc.

The parallelized data input sub-system 11 processes a table to determinehow to store it. For example, the parallelized data input sub-system 11divides the data set into a plurality of data partitions. For eachpartition, the parallelized data input sub-system 11 divides it into aplurality of data segments based on a segmenting factor. The segmentingfactor includes a variety of approaches divide a partition intosegments. For example, the segment factor indicates a number of recordsto include in a segment. As another example, the segmenting factorindicates a number of segments to include in a segment group. As anotherexample, the segmenting factor identifies how to segment a datapartition based on storage capabilities of the data store and processingsub-system. As a further example, the segmenting factor indicates howmany segments for a data partition based on a redundancy storageencoding scheme.

As an example of dividing a data partition into segments based on aredundancy storage encoding scheme, assume that it includes a 4 of 5encoding scheme (meaning any 4 of 5 encoded data elements can be used torecover the data). Based on these parameters, the parallelized datainput sub-system 11 divides a data partition into 5 segments: onecorresponding to each of the data elements).

The parallelized data input sub-system 11 restructures the plurality ofdata segments to produce restructured data segments. For example, theparallelized data input sub-system 11 restructures records of a firstdata segment of the plurality of data segments based on a key field ofthe plurality of data fields to produce a first restructured datasegment. The key field is common to the plurality of records. As aspecific example, the parallelized data input sub-system 11 restructuresa first data segment by dividing the first data segment into a pluralityof data slabs (e.g., columns of a segment of a partition of a table).Using one or more of the columns as a key, or keys, the parallelizeddata input sub-system 11 sorts the data slabs. The restructuring toproduce the data slabs is discussed in greater detail with reference toFIG. 4 and FIGS. 16-18 .

The parallelized data input sub-system 11 also generates storageinstructions regarding how sub-system 12 is to store the restructureddata segments for efficient processing of subsequently received queriesregarding the stored data. For example, the storage instructions includeone or more of: a naming scheme, a request to store, a memory resourcerequirement, a processing resource requirement, an expected accessfrequency level, an expected storage duration, a required maximum accesslatency time, and other requirements associated with storage,processing, and retrieval of data.

A designated computing device of the parallelized data store, retrieve,and/or process sub-system 12 receives the restructured data segments andthe storage instructions. The designated computing device (which israndomly selected, selected in a round robin manner, or by default)interprets the storage instructions to identify resources (e.g., itself,its components, other computing devices, and/or components thereof)within the computing device's storage cluster. The designated computingdevice then divides the restructured data segments of a segment group ofa partition of a table into segment divisions based on the identifiedresources and/or the storage instructions. The designated computingdevice then sends the segment divisions to the identified resources forstorage and subsequent processing in accordance with a query. Theoperation of the parallelized data store, retrieve, and/or processsub-system 12 is discussed in greater detail with reference to FIG. 6 .

The parallelized query and response sub-system 13 receives queriesregarding tables (e.g., data sets) and processes the queries prior tosending them to the parallelized data store, retrieve, and/or processsub-system 12 for execution. For example, the parallelized query andresponse sub-system 13 generates an initial query plan based on a dataprocessing request (e.g., a query) regarding a data set (e.g., thetables). Sub-system 13 optimizes the initial query plan based on one ormore of the storage instructions, the engaged resources, andoptimization functions to produce an optimized query plan.

For example, the parallelized query and response sub-system 13 receivesa specific query no. 1 regarding the data set no. 1 (e.g., a specifictable). The query is in a standard query format such as Open DatabaseConnectivity (ODBC), Java Database Connectivity (JDBC), and/or SPARK.The query is assigned to a node within the parallelized query andresponse sub-system 13 for processing. The assigned node identifies therelevant table, determines where and how it is stored, and determinesavailable nodes within the parallelized data store, retrieve, and/orprocess sub-system 12 for processing the query.

In addition, the assigned node parses the query to create an abstractsyntax tree. As a specific example, the assigned node converts an SQL(Standard Query Language) statement into a database instruction set. Theassigned node then validates the abstract syntax tree. If not valid, theassigned node generates a SQL exception, determines an appropriatecorrection, and repeats. When the abstract syntax tree is validated, theassigned node then creates an annotated abstract syntax tree. Theannotated abstract syntax tree includes the verified abstract syntaxtree plus annotations regarding column names, data type(s), dataaggregation or not, correlation or not, sub-query or not, and so on.

The assigned node then creates an initial query plan from the annotatedabstract syntax tree. The assigned node optimizes the initial query planusing a cost analysis function (e.g., processing time, processingresources, etc.) and/or other optimization functions. Having producedthe optimized query plan, the parallelized query and response sub-system13 sends the optimized query plan to the parallelized data store,retrieve, and/or process sub-system 12 for execution. The operation ofthe parallelized query and response sub-system 13 is discussed ingreater detail with reference to FIG. 5 .

The parallelized data store, retrieve, and/or process sub-system 12executes the optimized query plan to produce resultants and sends theresultants to the parallelized query and response sub-system 13. Withinthe parallelized data store, retrieve, and/or process sub-system 12, acomputing device is designated as a primary device for the query plan(e.g., optimized query plan) and receives it. The primary deviceprocesses the query plan to identify nodes within the parallelized datastore, retrieve, and/or process sub-system 12 for processing the queryplan. The primary device then sends appropriate portions of the queryplan to the identified nodes for execution. The primary device receivesresponses from the identified nodes and processes them in accordancewith the query plan.

The primary device of the parallelized data store, retrieve, and/orprocess sub-system 12 provides the resulting response (e.g., resultants)to the assigned node of the parallelized query and response sub-system13. For example, the assigned node determines whether further processingis needed on the resulting response (e.g., joining, filtering, etc.). Ifnot, the assigned node outputs the resulting response as the response tothe query (e.g., a response for query no. 1 regarding data set no. 1).If, however, further processing is determined, the assigned node furtherprocesses the resulting response to produce the response to the query.Having received the resultants, the parallelized query and responsesub-system 13 creates a response from the resultants for the dataprocessing request.

FIG. 2 is a schematic block diagram of an embodiment of theadministrative sub-system 15 of FIG. 1A that includes one or morecomputing devices 18-1 through 18-n. Each of the computing devicesexecutes an administrative processing function utilizing a correspondingadministrative processing of administrative processing 19-1 through 19-n(which includes a plurality of administrative operations) thatcoordinates system level operations of the database system. Eachcomputing device is coupled to an external network 17, or networks, andto the system communication resources 14 of FIG. 1A.

As will be described in greater detail with reference to one or moresubsequent figures, a computing device includes a plurality of nodes andeach node includes a plurality of processing core resources. Eachprocessing core resource is capable of executing at least a portion ofan administrative operation independently. This supports lock free andparallel execution of one or more administrative operations.

The administrative sub-system 15 functions to store metadata of the dataset described with reference to FIG. 1A. For example, the storingincludes generating the metadata to include one or more of an identifierof a stored table, the size of the stored table (e.g., bytes, number ofcolumns, number of rows, etc.), labels for key fields of data segments,a data type indicator, the data owner, access permissions, availablestorage resources, storage resource specifications, software foroperating the data processing, historical storage information, storagestatistics, stored data access statistics (e.g., frequency, time of day,accessing entity identifiers, etc.) and any other information associatedwith optimizing operation of the database system 10.

FIG. 3 is a schematic block diagram of an embodiment of theconfiguration sub-system 16 of FIG. 1A that includes one or morecomputing devices 18-1 through 18-n. Each of the computing devicesexecutes a configuration processing function 20-1 through 20-n (whichincludes a plurality of configuration operations) that coordinatessystem level configurations of the database system. Each computingdevice is coupled to the external network 17 of FIG. 2 , or networks,and to the system communication resources 14 of FIG. 1A.

FIG. 4 is a schematic block diagram of an embodiment of the parallelizeddata input sub-system 11 of FIG. 1A that includes a bulk data sub-system23 and a parallelized ingress sub-system 24. The bulk data sub-system 23includes a plurality of computing devices 18-1 through 18-n. A computingdevice includes a bulk data processing function (e.g., 27-1) forreceiving a table from a network storage system 21 (e.g., a server, acloud storage service, etc.) and processing it for storage as generallydiscussed with reference to FIG. 1A.

The parallelized ingress sub-system 24 includes a plurality of ingressdata sub-systems 25-1 through 25-p that each include a localcommunication resource of local communication resources 26-1 through26-p and a plurality of computing devices 18-1 through 18-n. A computingdevice executes an ingress data processing function (e.g., 28-1) toreceive streaming data regarding a table via a wide area network 22 andprocessing it for storage as generally discussed with reference to FIG.1A. With a plurality of ingress data sub-systems 25-1 through 25-p, datafrom a plurality of tables can be streamed into the database system 10at one time.

In general, the bulk data processing function is geared towardsreceiving data of a table in a bulk fashion (e.g., the table exists andis being retrieved as a whole, or portion thereof). The ingress dataprocessing function is geared towards receiving streaming data from oneor more data sources (e.g., receive data of a table as the data is beinggenerated). For example, the ingress data processing function is gearedtowards receiving data from a plurality of machines in a factory in aperiodic or continual manner as the machines create the data.

FIG. 5 is a schematic block diagram of an embodiment of a parallelizedquery and results sub-system 13 that includes a plurality of computingdevices 18-1 through 18-n. Each of the computing devices executes aquery (Q) & response (R) processing function 33-1 through 33-n. Thecomputing devices are coupled to the wide area network 22 to receivequeries (e.g., query no. 1 regarding data set no. 1) regarding tablesand to provide responses to the queries (e.g., response for query no. 1regarding the data set no. 1). For example, a computing device (e.g.,18-1) receives a query, creates an initial query plan therefrom, andoptimizes it to produce an optimized plan. The computing device thensends components (e.g., one or more operations) of the optimized plan tothe parallelized data store, retrieve, &/or process sub-system 12.

Processing resources of the parallelized data store, retrieve, &/orprocess sub-system 12 processes the components of the optimized plan toproduce results components 32-1 through 32-n. The computing device ofthe Q&R sub-system 13 processes the result components to produce a queryresponse.

The Q&R sub-system 13 allows for multiple queries regarding one or moretables to be processed concurrently. For example, a set of processingcore resources of a computing device (e.g., one or more processing coreresources) processes a first query and a second set of processing coreresources of the computing device (or a different computing device)processes a second query.

As will be described in greater detail with reference to one or moresubsequent figures, a computing device includes a plurality of nodes andeach node includes multiple processing core resources such that aplurality of computing devices includes pluralities of multipleprocessing core resources A processing core resource of the pluralitiesof multiple processing core resources generates the optimized query planand other processing core resources of the pluralities of multipleprocessing core resources generates other optimized query plans forother data processing requests. Each processing core resource is capableof executing at least a portion of the Q & R function. In an embodiment,a plurality of processing core resources of one or more nodes executesthe Q & R function to produce a response to a query. The processing coreresource is discussed in greater detail with reference to FIG. 13 .

FIG. 6 is a schematic block diagram of an embodiment of a parallelizeddata store, retrieve, and/or process sub-system 12 that includes aplurality of computing devices, where each computing device includes aplurality of nodes and each node includes multiple processing coreresources. Each processing core resource is capable of executing atleast a portion of the function of the parallelized data store,retrieve, and/or process sub-system 12. The plurality of computingdevices is arranged into a plurality of storage clusters. Each storagecluster includes a number of computing devices.

In an embodiment, the parallelized data store, retrieve, and/or processsub-system 12 includes a plurality of storage clusters 35-1 through35-z. Each storage cluster includes a corresponding local communicationresource 26-1 through 26-z and a number of computing devices 18-1through 18-5. Each computing device executes an input, output, andprocessing (IO &P) processing function 34-1 through 34-5 to store andprocess data.

The number of computing devices in a storage cluster corresponds to thenumber of segments (e.g., a segment group) in which a data partitionedis divided. For example, if a data partition is divided into fivesegments, a storage cluster includes five computing devices. As anotherexample, if the data is divided into eight segments, then there areeight computing devices in the storage clusters.

To store a segment group of segments 29 within a storage cluster, adesignated computing device of the storage cluster interprets storageinstructions to identify computing devices (and/or processing coreresources thereof) for storing the segments to produce identifiedengaged resources. The designated computing device is selected by arandom selection, a default selection, a round-robin selection, or anyother mechanism for selection.

The designated computing device sends a segment to each computing devicein the storage cluster, including itself. Each of the computing devicesstores their segment of the segment group. As an example, five segments29 of a segment group are stored by five computing devices of storagecluster 35-1. The first computing device 18-1-1 stores a first segmentof the segment group; a second computing device 18-2-1 stores a secondsegment of the segment group; and so on. With the segments stored, thecomputing devices are able to process queries (e.g., query componentsfrom the Q&R sub-system 13) and produce appropriate result components.

While storage cluster 35-1 is storing and/or processing a segment group,the other storage clusters 35-2 through 35-n are storing and/orprocessing other segment groups. For example, a table is partitionedinto three segment groups. Three storage clusters store and/or processthe three segment groups independently. As another example, four tablesare independently storage and/or processed by one or more storageclusters. As yet another example, storage cluster 35-1 is storing and/orprocessing a second segment group while it is storing/or and processinga first segment group.

FIG. 7 is a schematic block diagram of an embodiment of a computingdevice 18 that includes a plurality of nodes 37-1 through 37-4 coupledto a computing device controller hub 36. The computing device controllerhub 36 includes one or more of a chipset, a quick path interconnect(QPI), and an ultra path interconnection (UPI). Each node 37-1 through37-4 includes a central processing module 39-1 through 39-4, a mainmemory 40-1 through 40-4 (e.g., volatile memory), a disk memory 38-1through 38-4 (non-volatile memory), and a network connection 41-1through 41-4. In an alternate configuration, the nodes share a networkconnection, which is coupled to the computing device controller hub 36or to one of the nodes as illustrated in subsequent figures.

In an embodiment, each node is capable of operating independently of theother nodes. This allows for large scale parallel operation of a queryrequest, which significantly reduces processing time for such queries.In another embodiment, one or more node function as co-processors toshare processing requirements of a particular function, or functions.

FIG. 8 is a schematic block diagram of another embodiment of a computingdevice is similar to the computing device of FIG. 7 with an exceptionthat it includes a single network connection 41, which is coupled to thecomputing device controller hub 36. As such, each node coordinates withthe computing device controller hub to transmit or receive data via thenetwork connection.

FIG. 9 is a schematic block diagram of another embodiment of a computingdevice is similar to the computing device of FIG. 7 with an exceptionthat it includes a single network connection 41, which is coupled to acentral processing module of a node (e.g., to central processing module39-1 of node 37-1). As such, each node coordinates with the centralprocessing module via the computing device controller hub 36 to transmitor receive data via the network connection.

FIG. 10 is a schematic block diagram of an embodiment of a node 37 ofcomputing device 18. The node 37 includes the central processing module39, the main memory 40, the disk memory 38, and the network connection41. The main memory 40 includes read only memory (RAM) and/or other formof volatile memory for storage of data and/or operational instructionsof applications and/or of the operating system. The central processingmodule 39 includes a plurality of processing modules 44-1 through 44-nand an associated one or more cache memory 45. A processing module is asdefined at the end of the detailed description.

The disk memory 38 includes a plurality of memory interface modules 43-1through 43-n and a plurality of memory devices 42-1 through 42-n (e.g.,non-volatile memory). The memory devices 42-1 through 42-n include, butare not limited to, solid state memory, disk drive memory, cloud storagememory, and other non-volatile memory. For each type of memory device, adifferent memory interface module 43-1 through 43-n is used. Forexample, solid state memory uses a standard, or serial, ATA (SATA),variation, or extension thereof, as its memory interface. As anotherexample, disk drive memory devices use a small computer system interface(SCSI), variation, or extension thereof, as its memory interface.

In an embodiment, the disk memory 38 includes a plurality of solid statememory devices and corresponding memory interface modules. In anotherembodiment, the disk memory 38 includes a plurality of solid statememory devices, a plurality of disk memories, and corresponding memoryinterface modules.

The network connection 41 includes a plurality of network interfacemodules 46-1 through 46-n and a plurality of network cards 47-1 through47-n. A network card includes a wireless LAN (WLAN) device (e.g., anIEEE 802.11n or another protocol), a LAN device (e.g., Ethernet), acellular device (e.g., CDMA), etc. The corresponding network interfacemodules 46-1 through 46-n include a software driver for thecorresponding network card and a physical connection that couples thenetwork card to the central processing module 39 or other component(s)of the node.

The connections between the central processing module 39, the mainmemory 40, the disk memory 38, and the network connection 41 may beimplemented in a variety of ways. For example, the connections are madethrough a node controller (e.g., a local version of the computing devicecontroller hub 36). As another example, the connections are made throughthe computing device controller hub 36.

FIG. 11 is a schematic block diagram of an embodiment of a node 37 of acomputing device 18 that is similar to the node of FIG. 10 , with adifference in the network connection. In this embodiment, the node 37includes a single network interface module 46 and a correspondingnetwork card 47 configuration.

FIG. 12 is a schematic block diagram of an embodiment of a node 37 of acomputing device 18 that is similar to the node of FIG. 10 , with adifference in the network connection. In this embodiment, the node 37connects to a network connection via the computing device controller hub36.

FIG. 13 is a schematic block diagram of another embodiment of a node 37of computing device 18 that includes processing core resources 48-1through 48-n, a memory device (MD) bus 49, a processing module (PM) bus50, a main memory 40 and a network connection 41. The network connection41 includes the network card 47 and the network interface module 46 ofFIG. 10 . Each processing core resource 48 includes a correspondingprocessing module 44-1 through 44-n, a corresponding memory interfacemodule 43-1 through 43-n, a corresponding memory device 42-1 through42-n, and a corresponding cache memory 45-1 through 45-n. In thisconfiguration, each processing core resource can operate independentlyof the other processing core resources. This further supports increasedparallel operation of database functions to further reduce executiontime.

The main memory 40 is divided into a computing device (CD) 56 sectionand a database (DB) 51 section. The database section includes a databaseoperating system (OS) area 52, a disk area 53, a network area 54, and ageneral area 55. The computing device section includes a computingdevice operating system (OS) area 57 and a general area 58. Note thateach section could include more or less allocated areas for varioustasks being executed by the database system.

In general, the database OS 52 allocates main memory for databaseoperations. Once allocated, the computing device OS 57 cannot accessthat portion of the main memory 40. This supports lock free andindependent parallel execution of one or more operations.

FIG. 14 is a schematic block diagram of an embodiment of operatingsystems of a computing device 18. The computing device 18 includes acomputer operating system 60 and a database overriding operating system(DB OS) 61. The computer OS 60 includes process management 62, filesystem management 63, device management 64, memory management 66, andsecurity 65. The processing management 62 generally includes processscheduling 67 and inter-process communication and synchronization 68. Ingeneral, the computer OS 60 is a conventional operating system used by avariety of types of computing devices. For example, the computeroperating system is a personal computer operating system, a serveroperating system, a tablet operating system, a cell phone operatingsystem, etc.

The database overriding operating system (DB OS) 61 includes custom DBdevice management 69, custom DB process management 70 (e.g., processscheduling and/or inter-process communication & synchronization), customDB file system management 71, custom DB memory management 72, and/orcustom security 73. In general, the database overriding OS 61 provideshardware components of a node for more direct access to memory, moredirect access to a network connection, improved independency, improveddata storage, improved data retrieval, and/or improved data processingthan the computing device OS.

In an example of operation, the database overriding OS 61 controls whichoperating system, or portions thereof, operate with each node and/orcomputing device controller hub of a computing device (e.g., via OSselect 75-1 through 75-n when communicating with nodes 37-1 through 37-nand via OS select 75-m when communicating with the computing devicecontroller hub 36). For example, device management of a node issupported by the computer operating system, while process management,memory management, and file system management are supported by thedatabase overriding operating system. To override the computer OS, thedatabase overriding OS provides instructions to the computer OSregarding which management tasks will be controlled by the databaseoverriding OS. The database overriding OS also provides notification tothe computer OS as to which sections of the main memory it is reservingexclusively for one or more database functions, operations, and/ortasks. One or more examples of the database overriding operating systemare provided in subsequent figures.

FIGS. 15-23 are schematic block diagrams of an example of processing atable or data set for storage in the database system 10. FIG. 15illustrates an example of a data set or table that includes 32 columnsand 80 rows, or records, that is received by the parallelized datainput-subsystem. This is a very small table, but is sufficient forillustrating one or more concepts regarding one or more aspects of adatabase system. The table is representative of a variety of dataranging from insurance data, to financial data, to employee data, tomedical data, and so on.

FIG. 16 illustrates an example of the parallelized data input-subsystemdividing the data set into two partitions. Each of the data partitionsincludes 40 rows, or records, of the data set. In another example, theparallelized data input-subsystem divides the data set into more thantwo partitions. In yet another example, the parallelized datainput-subsystem divides the data set into many partitions and at leasttwo of the partitions have a different number of rows.

FIG. 17 illustrates an example of the parallelized data input-subsystemdividing a data partition into a plurality of segments to form a segmentgroup. The number of segments in a segment group is a function of thedata redundancy encoding. In this example, the data redundancy encodingis single parity encoding from four data pieces; thus, five segments arecreated. In another example, the data redundancy encoding is a twoparity encoding from four data pieces; thus, six segments are created.In yet another example, the data redundancy encoding is single parityencoding from seven data pieces; thus, eight segments are created.

FIG. 18 illustrates an example of data for segment 1 of the segments ofFIG. 17 . The segment is in a raw form since it has not yet been keycolumn sorted. As shown, segment 1 includes 8 rows and 32 columns. Thethird column is selected as the key column and the other columns storedvarious pieces of information for a given row (i.e., a record). The keycolumn may be selected in a variety of ways. For example, the key columnis selected based on a type of query (e.g., a query regarding a year,where a data column is selected as the key column). As another example,the key column is selected in accordance with a received input commandthat identified the key column. As yet another example, the key columnis selected as a default key column (e.g., a date column, an ID column,etc.)

As an example, the table is regarding a fleet of vehicles. Each rowrepresents data regarding a unique vehicle. The first column stores avehicle ID, the second column stores make and model information of thevehicle. The third column stores data as to whether the vehicle is on oroff. The remaining columns store data regarding the operation of thevehicle such as mileage, gas level, oil level, maintenance information,routes taken, etc.

With the third column selected as the key column, the other columns ofthe segment are to be sorted based on the key column. Prior to sorted,the columns are separated to form data slabs. As such, one column isseparated out to form one data slab.

FIG. 19 illustrates an example of the parallelized data input-subsystemdividing segment 1 of FIG. 18 into a plurality of data slabs. A dataslab is a column of segment 1. In this figure, the data of the dataslabs has not been sorted. Once the columns have been separated intodata slabs, each data slab is sorted based on the key column. Note thatmore than one key column may be selected and used to sort the data slabsbased on two or more other columns.

FIG. 20 illustrates an example of the parallelized data input-subsystemsorting the each of the data slabs based on the key column. In thisexample, the data slabs are sorted based on the third column whichincludes data of “on” or “off”. The rows of a data slab are rearrangedbased on the key column to produce a sorted data slab. Each segment ofthe segment group is divided into similar data slabs and sorted by thesame key column to produce sorted data slabs.

FIG. 21 illustrates an example of each segment of the segment groupsorted into sorted data slabs. The similarity of data from segment tosegment is for the convenience of illustration. Note that each segmenthas its own data, which may or may not be similar to the data in theother sections.

FIG. 22 illustrates an example of a segment structure for a segment ofthe segment group. The segment structure for a segment includes the data& parity section, a manifest section, one or more index sections, and astatistics section. The segment structure represents a storage mappingof the data (e.g., data slabs and parity data) of a segment andassociated data (e.g., metadata, statistics, key column(s), etc.)regarding the data of the segment. The sorted data slabs of FIG. 16 ofthe segment are stored in the data & parity section of the segmentstructure. The sorted data slabs are stored in the data & parity sectionin a compressed format or as raw data (i.e., non-compressed format).Note that a segment structure has a particular data size (e.g., 32Giga-Bytes) and data is stored within in coding block sizes (e.g., 4Kilo-Bytes).

Before the sorted data slabs are stored in the data & parity section, orconcurrently with storing in the data & parity section, the sorted dataslabs of a segment are redundancy encoded. The redundancy encoding maybe done in a variety of ways. For example, the redundancy encoding is inaccordance with RAID 5, RAID 6, or RAID 10. As another example, theredundancy encoding is a form of forward error encoding (e.g., ReedSolomon, Trellis, etc.). An example of redundancy encoding is discussedin greater detail with reference to one or more of FIGS. 29-36 .

The manifest section stores metadata regarding the sorted data slabs.The metadata includes one or more of, but is not limited to, descriptivemetadata, structural metadata, and/or administrative metadata.Descriptive metadata includes one or more of, but is not limited to,information regarding data such as name, an abstract, keywords, author,etc. Structural metadata includes one or more of, but is not limited to,structural features of the data such as page size, page ordering,formatting, compression information, redundancy encoding information,logical addressing information, physical addressing information,physical to logical addressing information, etc. Administrative metadataincludes one or more of, but is not limited to, information that aids inmanaging data such as file type, access privileges, rights management,preservation of the data, etc.

The key column is stored in an index section. For example, a first keycolumn is stored in index #0. If a second key column exists, it isstored in index #1. As such, for each key column, it is stored in itsown index section. Alternatively, one or more key columns are stored ina single index section.

The statistics section stores statistical information regarding thesegment and/or the segment group. The statistical information includesone or more of, but is not limited, to number of rows (e.g., datavalues) in one or more of the sorted data slabs, average length of oneor more of the sorted data slabs, average row size (e.g., average sizeof a data value), etc. The statistical information includes informationregarding raw data slabs, raw parity data, and/or compressed data slabsand parity data.

FIG. 23 illustrates the segment structures for each segment of a segmentgroup having five segments. Each segment includes a data & paritysection, a manifest section, one or more index sections, and a statisticsection. Each segment is targeted for storage in a different computingdevice of a storage cluster. The number of segments in the segment groupcorresponds to the number of computing devices in a storage cluster. Inthis example, there are five computing devices in a storage cluster.Other examples include more or less than five computing devices in astorage cluster.

FIGS. 24A-24K illustrate various embodiments of a node 37 of a computingdevice 18 that is operable to implement a segment scheduler module 2410.The embodiments illustrated in 24A-24K can be utilized to implement someor all of the plurality of nodes 37 of some or all computing devices18-1-18-n, for example, of the of the parallelized data store, retrieve,and/or process sub-system 12, and/or of the parallelized query andresults sub-system 13. The embodiments of node 37 discussed inconjunction with FIGS. 24A-24K can be utilized to implement any othernodes 37 of database system 10 discussed herein. The embodiments of node37 illustrated in FIGS. 24A-24K are operable to schedule retrievaland/or processing of a plurality of segments required for execution ofone or more queries over a plurality of sequential time slices. Inparticular, the retrieval and/or processing of segments can be scheduledbased on maximizing and/or otherwise optimizing drive utilization of aplurality of drives storing the plurality of segments.

As illustrated in FIG. 24A, a node 37 can include segment storage 2442that includes plurality of M memory drives 2440-1-2440-M. Differentnodes 37 can include the same or different number of memory drives. Someor all memory of each memory drive 2440 can be designated for storage ofa plurality of segments 2445. Different memory drives 2440-1-2440-M canstore the same or different number of segments. For example, asillustrated in FIG. 24A, memory drive 2440-1 can store X segments thatinclude segment 2445-1-1-2445-1-X; memory drive 2440-2 can store Ysegments that include segment 2445-2-1-2445-2-Y; and memory drive M canstore Z segments that include segment 2445-M-1-2445-M-Z. While thesegments are labeled with sequential numbers in FIG. 24A in each memorydrive, the set of segments stored by each memory drive 2440 cancorrespond to sequential or non-sequential partitions of data from thesame or different tables and/or same or different datasets of thedatabase system 10.

The segments stored by a memory drive 2440 can correspond to thesegments discussed in conjunction with FIGS. 15-23 , for example, wherethe segments are generated and stored in conjunction with a redundancystorage encoding scheme as discussed in conjunction with FIGS. 15-23 .Alternatively, the segments stored by memory devices 2440 as discussedherein can correspond to other data that are not generated inconjunction with the redundancy storage encoding scheme discussed inconjunction with FIGS. 15-23 . For example, some or all segments caninclude and/or be processed to recover a subset of rows of one or moretables; a subset of columns of one or more tables; a set of data slabsof one or more tables and/or one or more other data sets; a set of datapartitions of one or more tables and/or one or more other data sets;and/or other portions of data stored by the database system 10 asdiscussed herein. As discussed herein, each data segment can indicate aparticular subset of rows of a particular table, where a subset offields and/or columns or an entirety of fields and/or columns of eachrow in the particular subset of rows is included in the segment. In someembodiments, each segment of the node 37 is stored in exactly one memorydrive 2440-1-2440-M. In some embodiments, each segment of the databasesystem 10 is further stored in exactly one memory drive 2440 of exactlyone node 37.

Each memory drive 2440 can be implemented by one or more memory devicessuch as one or more solid state memory devices and/or disk memories.Different memory drives 2440-1-2440-M can be implemented by the same ordifferent one of more memory devices, and/or can be implemented by thesame or different types of one or more distinct memory devices.

In some embodiments, some or all memory drives 2440 of a node 37 areimplemented by utilizing disk memory 38 of the node 37 and/or mainmemory 40 of the node 37. For example, some or all memory drives2440-1-2440-M of a node 37 can each be implemented by a designatedportion of a memory device 42 of the node 37, where a single memorydevice 42 includes multiple memory drives 2440. As another example, someor all memory drives 2440 of a node 37 is implemented by its own memorydevice 42 of the node 37, where some or all memory devices 42-1-42-neach implement one memory device 2440. As another example, some or allmemory drive 2440 of a node 37 can be implemented by utilizing multiplememory devices 42 of the node. Alternatively, some or all memory drives2440 of a node 37 can be implemented utilizing other memory resourcesand/or additional memory devices of the node 37.

In some embodiments, all of the memory drives 2440-1-2440-M of aparticular node 37 are integrated within and/or accessible via storageresources of the particular node 37, such as disk memory 38 and/or mainmemory 40 of the node 37. In such cases, each of the plurality of nodes37 of one or more computing devices 18 can include and/or accesssegments stored by their own designated set of memory drives 2440, forexample, where each memory device is owned by and/or accessible byexactly one corresponding node 37. In other embodiments, some memorydrives 2440 are accessible by multiple nodes 37. In such cases, one ormore memory drives 2440 of implemented by a particular node 37 can beaccessed by other nodes 37, for example, where some or all nodes 37 in acomputing device 18 can access one or more memory drives 2440 of some orall other nodes 37 in the same computing device. In some cases, othernodes only access segments from one or more memory drives of aparticular node's memory resources to facilitate recovery of virtualsegments being processed by the other nodes, as discussed in furtherdetail in conjunction with FIGS. 25A-25L. In some embodiments, one ormore memory drives 2440 can be implemented utilizing shared resources ofmultiple nodes of the same computing device 18. In some embodiments, oneor more memory drives 2440 can be accessible by multiple nodes 37 ofmultiple different computing devices 18.

In such embodiments where memory drives 2440 are accessible by multiplenodes, the set of memory drives 2440-1-2440-M of a particular node 37can include all memory drives that the particular node 37 has access toand/or all memory drives that the particular node 37 utilizes toretrieve segments from storage in processing physical segments it owns,as discussed in conjunction with FIGS. 25A-25L and FIGS. 27A-27K. Thiscan include: at least one memory drive 2440 implemented utilizing thenode's own storage resources; at least one memory drive 2440 implementedutilizing at least one different node's own storage resources, whereeach different node is implemented by the same or different computingdevice 18; and/or at least one memory drive implemented utilizingadditional storage resources accessible by only the particular node 37or accessible by multiple nodes 37 including the particular node 37.

The node 37 can be operable to execute queries against the databasesystem by processing corresponding segments required for execution ofthe query. For example, as discussed previously, the node 37 can beimplemented within the parallelized query and response sub-system 13 forprocessing a portion of a particular query or the entirety of aparticular query. This can include identifying a segment set 2418 of aparticular query 2405, which indicates a proper subset of segmentsstored in the memory drives 2440-1-2440-M required to execute the query.As illustrated in FIG. 24A, a segment set 2418 can indicate a pluralityof segment identifiers or other information identifying thecorresponding segments, for example, enabling the node 37 to identifythe location of the corresponding node 37 in segment storage 2442 forretrieval.

The segment set 2418 of a particular query 2405 can include a set ofsegments that includes all fields of all rows required to execute theentirety of the query 2405. Alternatively, the segment set 2418 of aparticular query can include a proper subset of all segments thatinclude all fields of all rows required to execute the particular query,where the proper subset of segments includes all required fields of aproper subset of rows and/or a proper subset of all required fields ofsome or all rows. In such cases, the parallelized query and responsesub-system 13 can utilize a set of multiple nodes 37 of one or morecomputing devices 18 to execute a same query in accordance with a queryexecution plan as discussed previously, where each node 37 in the set ofmultiple nodes identifies its own segment set 2418 of segments requiredby the query that are accessible by the node. The union of segment sets2418 across the set of multiple nodes 37 executing the same query canincludes all segments required to execute the same query. Furthermore,the plurality of segment sets 2418 of the set of multiple nodes 37 canbe mutually exclusive to ensure that no same segments are processed bymultiple nodes in their parallelized execution of the query. Thisassignment of particular segments to particular nodes to facilitateexecution of a query in parallel is discussed in further detail inconjunction with 27A-27K. In some cases, the segment set 2418 for aparticular query can be received and/or indicated in a request toexecute the query, can determined based on the domain of the queryand/or based on tables indicated in the query. The segment set 2418 candetermined independently by a node 37, in isolation without globalcoordination, based on the information indicated by the correspondingquery, as discussed in conjunction with 27A-27K. The retrieval ofsegments to across multiple nodes to execute a query can correspond tonodes implemented in conjunction with an IO level of a query executionplan utilized to execute the entirety of the query, as discussed inconjunction with FIGS. 28A-28O.

As used herein, execution of a query by a particular node 37 cancorrespond to the execution of the portion of the query 2405 assigned tothe particular node, for example, by utilizing the particular node'sdetermined segment set 2418 of the query. The portion of the query 2405assigned to the node for execution and/or otherwise determined by thenode for execution can be indicated and/or determined as operator data2416 of the query 2405, which can indicate one or more operators of thequery to be performed by the node 37 utilizing the corresponding segmentset 2418. The portion of the query assigned to the node can include alloperators of the query, where the entire query is performed by the nodeon a subset of required rows. For example, a resultant generated by aparticular node's full execution of a query via retrieval and/orprocessing of the node's entire segment set 2418 may correspond to onlya portion of the entire query result, such as a subset of rows in afinal result set, where other nodes generate their own resultants viatheir own segment set 2418 to generate other portions of the fullresultant of the query. In such embodiments, a plurality of nodes canfully execute queries on portions of the data independently parallel,where resultants generate by each of the plurality of nodes can begathered into a final result of the query.

The portion of the query assigned to the node can include alternativelyinclude only a proper subset of operators of the query, where the entirequery is performed by the node on a subset of required rows. Forexample, the resultant generated by a particular node's full executionof a query via retrieval and processing of the node's entire segment set2418 may correspond to a plurality of rows that need to be furtherfiltered, aggregated, and/or processed via one or more other node'sexecution of the query. Thus execution of the query by the node, as usedherein, can correspond to processing all segments of the segment set ofthe query in accordance with a subset of operators required to executethe query, where different nodes are assigned for processing ofdifferent operators of the query to facilitate full execution of thequery via a query execution plan of multiple levels, as discussed infurther detail in conjunction with FIGS. 28A-28O.

For example, the resultant generated by the particular node's fullexecution of the query is sent to and/or accessible by another node inthe set of multiple nodes executing the query in their own execution ofthe query. As a particular example, one nodes' execution of a particularquery can include retrieving all segments in the segment set and sendingthe required fields of the raw rows included in the segments of thesegment set, or other raw data included in the segments of the segmentset to another node responsible for performing query operators such asfiltering and/or aggregation of the set of rows. For example, the nodemay only be responsible for performing reads of the data required toexecute the query, where operators are to be performed on this data byone or more other nodes to ultimately fully execute the query. In suchembodiments, this other node may not have and/or may not utilize theirown set of memory drives 2440-1-2440-M, where these other nodes utilizeresultants outputted by the particular node and/or at least one othernode rather than utilizing raw rows of segments retrieved from memorydrives 2440-1-2440-M. For example, this other node can correspond to anode implementing an inner level or root level of a query execution planof FIG. 28A. In such embodiments, a plurality of nodes can executeassigned subsets of query operators in series, where a resultantgenerated by one node and/or resultants generated by multiple nodesperforming the same operators on different distinct rows and/ordifferent distinct subsets of previous resultants in parallel are sentto another node, where the another node utilizes the resultantsgenerated by this one of more nodes as input to generate its ownresultant based on its assigned subset of query operators of the query.

As used herein, partial execution of a query by a particular node 37 cancorrespond to retrieval and/or processing of a subset of the node'sdetermined segment set 2418 of the query, and/or processing of a propersubset of the node's assigned operators of the query on some or allsegments of the determined segment set 2418. Thus, a node's fullexecution of a particular query is facilitated via a plurality ofpartial executions of the query, where each partial execution includespartially and/or fully processing one or more segments in accordancewith the portion of the query assigned to the node. The node's fullexecution of the query can include generating a plurality of partialresultants that render a resultant of the node's execution of itsportion of the query. As used herein, a query resultant generated by aparticular node 37 is not necessarily a final resultant of the query. Anode's resultant can be utilized as input by other nodes to furtherprocess the query via other operators of the query.

If the node is instead receiving resultants from other nodes, forexample, by receiving the full set of partial resultants from each othernode at once or receiving each of the set of partial resultants one at atime as they are generated by each of a set of other nodes, the node'spartial execution of the query can correspond to performing its assignedsubset of query operations upon a corresponding one or more receivedpartial resultants. A node's full execution of the query can correspondto generating its own plurality partial resultants by utilizing theplurality of full or partial resultants received from all of a set ofother nodes that forwarded their own resultants to the node.

The node 37 discussed in conjunction with FIGS. 24A-24K performs partialexecutions upon segments to execute queries, and for example, does notreceive resultants from other nodes that are utilized as input inprocessing queries. The node 37 discussed in conjunction with FIGS.24A-24K can be operable to forward or send its partial and/or fullresultants generated via the plurality of partial executions to one ormore other nodes for processing via other operators of the query.

As illustrated in FIG. 24A, a node's performance of the plurality ofpartial executions of a query to ultimately generate its resultant forthe query can be achieved by utilizing a segment processing module 2430of the node 37. The segment processing module 2430 can be implemented byutilizing one or more of the processing core resources 48-1-48-n of thenode 37, as further discussed in conjunction with FIGS. 24B and 24C.Alternatively, any other one or more processing modules included in thenode 37 and/or available to the node 37 can be utilized by the node 37to facilitate the performance of the plurality of partial executions.

A plurality of partial resultants can each be generated based onprocessing, via the segment processing module 2430, one or moreparticular segments. The query's resultant, corresponding to output ofthe node's execution of the query, is generated by the segmentprocessing module 2430 based on: performing a union upon the pluralityof partial resultants; gathering the plurality of partial resultants;combining the plurality of partial resultants; aggregating the pluralityof partial resultants; and/or processing the plurality of partialresultants via one or more additional operators of the query. Some orall of the plurality of partial executions of the query required tofulfill the node's execution of the query can be performed in sequence,for example, where the node 37 processes each of the plurality ofsegments of the query one at a time in accordance with the operator data2416 to generate a corresponding plurality of partial resultants. Someor all of the plurality of partial executions of a query required tofulfill the node's execution of the query can be facilitated by the nodeconcurrently, for example, where different parallel processing threadsof the same or different processing core resource 48 of the node processdifferent segments in accordance with the assigned operators of thequery.

In particular, as illustrated in FIG. 24A, the processing of a segmentas one or more corresponding partial executions of a given query caninclude retrieving the segment from segment storage 2442. The partialexecution of the query for the given segment can include only theretrieval of the segment from segment storage, where the query resultantgenerated by the node includes the raw segments in the segment setand/or raw rows extracted from the retrieved segments in the segmentset. For example, the node's operator data indicates only row readoperations of the query. Alternatively, the same or different partialexecution of the query for the given segment can include additionalprocessing of the segment and/or the raw rows of the segment, onceretrieved from segment storage 2442, in accordance with the operatordata 2416.

Execution of a particular query 2405 by segment processing module 2430can be performed over a span of time. As used herein, a time slice cancorrespond to a temporal period of time. A set of sequential time slicescan include multiple, consecutive time slices of the same or differenttemporal length. In a given time slice, at least one partial executionof at least one query can be initiated by the node 37 and/or can befacilitated in its entirety by the node 37. Thus, a query's execution bythe node 37 can be performed across a corresponding set of sequentialtime slices, where some or all of the plurality of sequential timeslices can include initiation of at least one least one partialexecution of the query. The set of sequential time slices for a givenquery can begin with a first time slice corresponding to initiation of afirst partial execution, such as the first time a segment in the segmentset is retrieved. The set of sequential time slices for a given querycan end with a last time slice, corresponding to the time slice where afinal one of the plurality of partial executions is initiated and/orcompleted, and/or corresponding to the time slice where the resultant ofthe query is generated by the node from the plurality of partialresultants.

A partial execution can be completed in the same time slice in which itwas initiated, or can be performed across a sequential subset of the setof sequential time slices. In some cases, at least one of a query's setof sequential time slices does not include any initiation or any portionof facilitation of execution of any partial execution of the query,where some time slices are “skipped” in initiating or facilitating aquery's execution. In some cases, at least one of the set of sequentialtime slices includes initiation of and/or facilitation of at least aportion of multiple partial executions of the same query, wheredifferent parallel threads are utilized to concurrently perform thesemultiple partial executions of the same query in parallel within one ormore same time slices.

As illustrated in FIG. 24A, a node 37 can be assigned and/or canotherwise determine a query set 2415 for execution, which can include aset of queries 2405-1-2405-N for ordered and/or unordered execution bythe node 37, in series or concurrently. Each query can include its owncorresponding segment set of the same or different number of segments.Some or all segment sets of different queries in the query set 2415 canhave non-null intersections in response to their corresponding queriesrequiring access to the same tables and/or sets of rows. Some segmentsets of different queries in the query set 2415 can be identical. Somesegment sets of different queries in the query set 2415 can have nullintersections.

The concurrent execution of the multiple queries can be achieved via thesegment processing module 2430, where different parallel processingthreads of the segment processing module 2430 can perform partialexecutions of different queries concurrently, for example, as discussedin conjunction with FIGS. 26A-26C. Each query in the query set can beexecuted in its own set of sequential time slices, where differentqueries in the query set can have overlapping or non-overlapping sets ofsequential time slices. Within a plurality of sequential time slices,execution of some or all of the set of queries 2405-1-2405-N can befacilitated by the segment processing module 2430 to ultimately generatea corresponding plurality of query resultants 2432-1-2432-N, where eachone of the plurality of query resultants 2432 is based on a set ofpartial results generated for the corresponding one of the plurality ofqueries by processing the corresponding set of segments in the query'ssegment set 2418. Thus, the plurality of sequential time slices caninclude the plurality of sets of sequential time slices corresponding tothe plurality of queries in the query set 2415, where some or all of theplurality of sets of sequential times slices include overlapping timeslices or otherwise include overlapping temporal periods.

For example, consider two different queries 2405 in the query set 2415that includes a first query and a second query. The first query can beinitiated in a first time slice, and the second query can be initiatedin a second time slice, where the second time slice is after the firsttime slice. Execution of the first query can be completed in a thirdtime slice and execution of the second query can be completed in afourth time slice, where the third time slice is before, after, or thesame as the fourth time slice. The first query thus executes over afirst set of sequential time slices beginning with the first time sliceand ending with the third time slice. The second query executes over asecond set of sequential time slices beginning with the second timeslice and ending with the fourth time slice. Partial execution of thefirst query can be initiated and/or facilitated within every one of thefirst set of sequential time slices. Alternatively, at least one timeslice in the first set of time slices does not include initiation or anyportion of a partial execution of the first query, but does includeinitiation or at least one portion of at least one partial execution ofthe second query. Similarly, at least one time slice in the second setof time slices can include no initiation or no portion of a partialexecution of the second query, but can include initiation or at leastone portion of at least one partial execution of the first query. Atleast one time slice in the first set of sequential time slices andsecond set of sequential time slices can include initiation of and/or atleast one portion of a partial execution for the first query, and canfurther include initiation of and/or at least one portion of a partialexecution for the second query, for example, where these partialexecutions are facilitated in the same time slice via different parallelthreads of the segment processing module 2430.

New queries can be assigned, received, and/or determined for executionby the node 37, and can thus be added to the node's query set 2415overtime to generate updated query sets that include the new queries.For example, while at least one query in a prior query set 2415 is inthe process of being executed, a new query can be added to generate anupdated query set 2415, where the segment processing module 2430 canbeing executing the new query in the updated query set before or afterexecution of some or all queries in the prior query set have completed.

As illustrated in FIG. 24A, node 37 can further include a segmentscheduler module 2410. A node's segment scheduler module 2410 can beimplemented utilizing at least one processor and memory of the node 37.For example, the segment scheduler module can be implemented byutilizing one or more processing modules 44-1-44-n of central processingmodule 39 of the node 37; main memory 40 of the node 37, for exampleallocated for the computing device OS 57; and/or cache memory 45 of thenode 37. As a particular example, the process scheduling 67 of thecomputing device 18 that implements the node 37, implemented viacomputing device OS 57 of the node 37, can be utilized to implement thesegment scheduler module 2410. Alternatively, any other additionalprocessing and/or memory resources of the node and/or accessible to thenode can be utilized to implement the segment scheduler module 2410.

The segment scheduler module 2410 of a node 37 can locally store,access, or otherwise determine the query set 2415 of the node at anygiven time slice or otherwise at given points in time. The segmentscheduler module 2410 can facilitate scheduling of the plurality ofpartial executions of each of the plurality of queries in the query set2415 over the plurality of time slices by selecting which segments ofwhich queries will be processed in a given time slice. This can beaccomplished by utilizing a segment processing assignment module 2420 ofthe segment scheduler module 2410 and/or by otherwise utilizing at leastone processor and memory of the segment scheduler module 2410. Thesegment processing assignment module 2420 can select, for a givencurrent and/or upcoming time slice, at least one segment of at least onesegment set in the query set 2415 for retrieval from its correspondingmemory drive 2440, and/or for other processing in accordance withoperator data 2416. This can be indicated in segment processingselection data 2428 that is generated by the segment processingassignment module 2420, and the segment processing selection data 2428can be sent to and/or can otherwise be accessed by the segmentprocessing module 2430.

Thus, a plurality of segment processing selection data 2428 can begenerated by the segment processing assignment module 2420 for each of aplurality of sequential time slices, and/or can otherwise besequentially generated over time. The segment processing module 2430 canreceive this plurality of segment processing selection data 2428 insequence as it is generated over time, and can perform partialexecutions by performing the retrieval or other processing of thecorresponding segments indicated in the segment processing selectiondata 2428 in a corresponding plurality of sequential time slices.

As partial executions of queries are initiated and/or completed, thecorresponding segments in the query set can be flagged and/or otherwiseindicated as having their corresponding processing initiated and/orcompleted. When all segments of a given query have been fully processedin accordance with the operator data 2416 of the given query and/or whenthe resultant for the query is generated and/or sent to another node forprocessing, the query can be deemed as having been executed, and can beremoved from the query set and/or can otherwise be indicated in thequery set as having been completed. Alternatively, as each partialresultant is generated, it can be sent to another node for processing,for example, where the other node begins processing partial resultantsas they are received even if the entirety of partial resultants have notyet been generated by the node.

Time slices for which segment processing selection data indicatessegments for retrieval are not necessarily equal in length, where thesegment scheduler module does not necessarily request that new segmentsbe processed in regular fixed intervals. In some embodiments, thesegment processing selection data is generated in response todetermining that a currently executing query has completed a partialexecution and/or has otherwise completed retrieval and/or processing ofa segment previously indicated in previous segment processing selectiondata 2428. For example, a particular processing core resource 48,processing thread, and/or other processing resource allocated forexecution of a particular one of the set of queries in the query set canindicate that it has completed processing of at least one previouslyselected segment, and is thus ready to process a new segment, forexample, via a notification to the segment scheduler, via an update toquery set 2415 indicating completion of processing of the previoussegment, and/or via an indication that no segments of the query arecurrently being processed. A new segment of the query's segment set canbe selected by the segment processing assignment module 2420 in responseto determining that the previously selected segments of the query havecompleted processing. This mechanism of assigning segments forparticular queries in the query set as their corresponding processingresources are completed processing prior segments in their segment setcan dictate the plurality of sequential time slices as discussed herein,where a new time slice is initiated in response to determining to assigna new segment for processing of a query in response to one or morepreviously assigned segments have completed processing as completedpartial executions of the query. Note that, if multiple queries areready for a new segment, their requests for new segments may need to bequeued and/or otherwise divided across multiple sequential time slicesfor retrieval, as dictated by the segment scheduler module 2410.

In some embodiments, the segment processing selection data 2428 canfurther allocate processing resources of the segment processing module2430 for retrieval and/or processing of each particular segment and/orindicate which processing resources of the segment processing module2430 are utilized to retrieve and/or process each particular segment.Such embodiments are illustrated in FIGS. 24B and 24C. In particular,the segment processing module 2430 can be implemented by utilizing someor all of the processing core resources 48, where each partial executionis assigned to a processing core resource. Each processing core resource48 can initiate and/or perform one or more partial executions of one ormore corresponding segments of one or more corresponding queries in asingle time slice. For example, a single processing core resource 48 canfacilitate concurrent partial executions of the same or different queryin a single time slice by utilizing multiple parallel threads of theprocessing core resource 48. Alternatively, a single processing coreresource can be responsible for one or more partial executions ofexactly one query in a given time slice, and/or can be responsible forpartial execution of exactly one segment in a given time slice. Thesegment processing selection data 2428 for a given time slice canindicate a set of partial executions assigned to a set of differentprocessing core resources 48, where some or all processing coreresources 48 initiate or perform at least one of its own partialexecutions within a given time slice.

In the example illustrated in FIG. 24B, the segment processing selectiondata 2428 of a given time slice indicates that segment 3 be retrieved byprocessing core resource 48-2, for example, via its processing module44-2 and/or memory interface module 43-2. This can be based on thesegment processing assignment module 2420 selecting segment 3 andfurther selecting processing core resource 48-2. In some cases, asillustrated in FIG. 24B, the segment processing selection data 2428indicating selection of segment 3 for retrieval is sent directly toprocessing core resource 48-2, and not the other processing coreresources, in response to the segment processing assignment module 2420selecting processing core resource 48-2 for retrieval of segment 3. Inother embodiments, the segment processing module 2430 and/or a differentprocessing module of the node 37 can be responsible for allocation ofresources of the segment processing module for processing of segmentsindicated by incoming segment processing selection data 2428, where thesegment processing assignment module 2420 does not select whichprocessing core resource will be utilized for processing of selectedsegments.

In response to receiving the instruction to retrieve segment 3 asindicated by the segment processing selection data 2428, the processingcore resource 48-2 can determine segment 3 is stored in memory drive2440-2. For example, the segment processing selection data 2428 canindicate the location of segment 3 and/or can indicate segment 3 as anaddress or other location data in memory drive 2, for example, based onthe segment scheduling module 2410 utilizing location data indicated bythe segment identifier in the segment set 2418 and/or utilizing a lookuptable, metadata, or other information accessible locally by the node orotherwise accessible via the database system 10 that indicates storagelocations of particular rows of a query or otherwise indicates storagelocations of particular segments. Alternatively, the processing coreresource 48-2 itself can determine that segment 3 is stored in memorydevice 2 based on utilizing the segment identifier of segment 3indicated in the segment processing selection data 2428 and/or based onaccessing a storage location lookup table and/or segment storage mappinginformation.

In some embodiments, processing core resources are mapped to one or moreparticular memory drives 2440, and a processing core resource 48 isautomatically selected for retrieval and/or processing of a particularsegments based on the segment being stored in the one or more particularmemory drives 2440 mapped to the particular processing core resource.For example, each memory drive 2440 can be implemented utilizing some orall of a particular one of the set of memory device 42-1-42-n, whereeach of the set of memory devices 42-1-42-n is included in, assigned to,or utilized by a corresponding one of the set of processing coreresources 48-1-48-n as illustrated in FIG. 13 . In such cases, inresponse to selecting segment 3 for retrieval in the segment processingselection data 2428, processing core resource 48-2 can automatically beselected for retrieval of segment 3 in response to determining thatsegment 3 is stored in memory drive 2440-2 and further in response todetermining memory drive 2440-2 is implemented by and/or included inmemory device 42-2 that is mapped to processing core resource 48-2.

Once memory drive 2440-2 is identified, processing core resource 48-2can retrieve segment 3 from memory drive 2440-2. For example, processingcore resource 48-2 can send a retrieval request indicating segment 3 andcan retrieve segment 3 from the memory drive in response. In otherembodiments, the segment scheduling module 2410 itself can send requeststo memory drives indicating instructions to send the selected segmentsto segment processing module 2430 and/or to a selected processing coreresource 48 of segments processing module 2430 for processing. Inresponse to receiving a request for a segment from the processing coreresource and/or from the segment scheduler, the memory drive can sendthe requested segment to the requesting and/or indicated processing coreresource in response.

In some embodiments, the retrieval of the segment constitutes theentirety of partial execution of the segment, and/or other execution ofthe segment can be facilitated via a different processing core resource48 and/or a different node 37. However, the assigned core processingresource can facilitate the node's full processing of the segment inaccordance with the operator data 2416 of the corresponding query.

Such an embodiment is illustrated in FIG. 24C, where the processing coreresource generates a partial resultant for query 2 by processing segment5 in accordance with operator data 2416 of query 2. In such embodiments,the segment processing selection data 2428 or can indicate instructionsto process segment 5 in accordance with query 2, where the operator datafor query 2 is also sent to the processing core resource 48-2. Forexample, the operator data for query 2 can be sent to processing coreresource 48-2 only once, and the processing core resource 48-2 canutilize this operator data of query 2 in executing a plurality ofpartial executions for some or all of the segments in the segment setfor query 2. Alternatively, if the node serves to only retrieve segmentsin query segment sets and extract their raw data for processing by othernodes in accordance with the query, each processing core resource canprocess retrieved segments for any query in the same fashion byextracting the necessary rows or other raw data and/or routing thisextracted raw data to another node for further processing.

Furthermore, consider an example where the segment processing selectiondata 2428 of FIG. 24B occurs at one of the plurality of sequential timeslices t₀, and that the segment processing selection data 2428 of FIG.24C occurs at a later one of the plurality of sequential time slices t₁.Also assume that segment 3 was similarly processed to produce a partialresultant for query 2 in a similar fashion as segment 5 of FIG. 3 , andthat processing core resource 48-2 is assigned to facilitate some or allof the node 37's execution of query 2. As illustrated in FIG. 24C, theprocessing core resource 48-2 can send a notification to the segmentscheduler indicating that it has completed processing of segment 3 forquery 2, for example, in response to generating a partial resultant byprocessing segment 3. The segment scheduler, in response to determiningthat processing core resource 48-2 is ready to process a new segment forquery 2, can send the segment processing selection data 2428 at t₁indicating that the next segment selected to be processed for query 2 issegment 5.

Note that t₁ could be the time slice immediately following to in theplurality of sequential time slices, where no other segment processingselection data 2428 is generated by the segment processing assignmentmodule 2420 between the segment processing selection data 2428 at to andthe segment processing selection data 2428 at t₁. However, there mayhave been multiple other segment processing selection data 2428 that wasgenerated between t₀ and t₁ for other queries being executed by the sameor different processing core resources 48, for example, based on otherpartial resultants having been generated within this time frame forother queries of the query set, and new segments being assigned forprocessing these other queries by the segment processing assignmentmodule 2420 in response.

FIGS. 24D-24K illustrate embodiments where the segment scheduling module2410 implements the segment processing assignment module 2420 to selectsegments at particular time slices based on utilization data of theplurality of drives 1-M. A given query will addresses or otherwiserequires some subset of the segments stored in one or more memory drives2440 of segment storage 2442, but it can be unpredictable as to whichsegments will be required at any given point in time. Different queriesof the query set 2415 running on different, possibly overlapping,segments can create unpredictable read patterns.

Because the processing of a segment set to facilitate execution of acorresponding query can be performed in any order to achieve the sameresultant, and because the processing of a plurality of segment setsfacilitate concurrent execution of a corresponding set of queries canalso be performed in any order to achieve the same set of correspondingresultants, the ordering of segments for processing over time can beintelligently selected via the segment processing assignment module 2420to improve efficiency of retrieval of segments from segment storage2442. The segment scheduler can be operable to schedule segments withthe aim to fully utilize each memory drive at any given point of time,up to its maximum amount of throughput. In particular, to improve and/oroptimize retrieval efficiency of the segments in segment sets 2418 ofone or more queries in a query set 2415, the segment processingassignment module 2420 can select segments for processing based onselecting corresponding memory drives for retrieval that are currentlyunder-utilized. The selection of segments over time can be based onmaximizing the utilization of each of the set of memory drives2440-1-2440-M at any particular point in time, up to a maximumutilization threshold of each of the set of memory drives. Thismechanism of intelligently selecting segments based on maximizing driveutilization across a set of drives improves a node's efficiency inconcurrently executing queries. Furthermore, this mechanism can beapplied across some or all of a plurality of nodes 37 in a databasesystem 10 via implementation of segment processing assignment module2420 by some or all of the plurality of nodes can improve efficiency ofquery execution by the database system 10 as a whole.

In the examples discussed in conjunction with FIGS. 24D-24J, considerthe example query set 2415 illustrated in FIG. 24D. The query setincludes N queries that include queries 1, 2, and N. Query 1 has asegment set identifying a set of segments that includes segments 1, 2,and X. Query 2 has a segment set identifying a set of segments thatincludes segments 3, 5, and Y. Query N has a segment set identifying aset of segments that includes segments 2, 5, and Z. In this example,memory drive 2440-1 stores a set of segments that includes segments 1,2, and X; memory drive 2440-2 stores a set of segments that includessegments 3, 4, and Y; and memory drive M stores a set of segments thatincludes segments 5, 6, and Z. Segments 1 and Y have been retrieved orinitiated for retrieval for processing of queries 1 and 2, respectively,via previously being selected in segment processing selection data 2428generated previously for one or more prior time slices. Thisconfiguration of segments in the query set and stored in segment storagecan also extend to the examples illustrated in FIGS. 24B and 24C.

In the current time slice, the segment processing selection data 2428indicates selection of segment 3 for retrieval from memory drive 2440-2,for example, to facilitate corresponding partial execution of query 1 orquery 2. In some cases, retrieval of segment 3 can be utilized tofacilitate partial execution of both query 1 or query 2, where segment 3is retrieved from memory only once to satisfy partial execution of bothqueries and to generate the same or different partial resultant for eachquery, where the resultant is the same or different based on whether therespective queries have the same or different operator data 2416. Forexample, a selected one of the plurality of processing core resources48-1-48-n can be assigned to retrieve segment 3 and can further beassigned facilitate concurrent partial execution of both query 1 andquery 2 utilizing the single retrieval of segment 3 to generate thecorresponding partial resultants for query 1 and query 2.

Each memory drive 2440 can have a known and/or determined maximumutilization threshold indicating a maximum possible amount ofutilization of the drive and/or a desired level of utilization the driveshould be achieving at any given time in an optimal scenario. Forexample, the maximum utilization threshold can be based on a maximumpossible throughput of the memory drive for transmission of retrievedsegments, based on processing resources or maximum processingcapabilities of the drive, based on the type of memory device utilizedto implement the memory drive, based on average or maximum seek time tolocate segments within the drive, and/or based on other time and/orprocessing constraints to access and/or transmit requested segments.Different ones of the set of memory drives 2440-1-2440-M of a particularnode 37 can have the same or different corresponding maximum utilizationthresholds. In some cases, the maximum utilization threshold is measuredand/or estimated by the segment scheduler or other processing module ofthe node based on averaging and/or analyzing processing times and/orresource consumption utilized by the memory drives in historicalretrieval of prior segments over time.

At a given time, drive utilization data 2425 can be received and/orgenerated by the segment scheduling module 2410. The drive utilizationdata 2425 can include actual and/or estimated utilization levels of someor all of the plurality of memory drives for a current, recent, and/orupcoming one or more time slices. A memory drive's utilization levelscan correspond to or be based on a raw measurement or estimate ofthroughput of the memory drive, a raw measurement or estimate ofresource utilization of the memory drive, and/or a raw measurement orestimate of another metric indicating a level of utilization of thememory drive. A memory drive's utilization level can correspond to or bebased on an actual or estimated percentage or proportion of the drive'smaximum utilization threshold utilized currently, utilized recently,and/or expected to be utilized in one or more upcoming time slices.

The set of maximum utilization thresholds and the drive utilization datacan be utilized to determine an actual or estimated availableutilization level for some or all of the set of memory drives 2440, forexample, calculated based on a difference between the raw measurement orestimate for utilization of the drive and the maximum utilizationthreshold of the drive. This available utilization level can similarlycorrespond to an estimated amount of availability or actual amount ofavailability for one or more current, recent and/or upcoming timeslices. In some cases, the drive utilization data indicates this set ofcalculated available utilization levels.

The segment processing assignment module can select one or more memorydrives to be accessed in the current or next upcoming time slice, asdictated in the segment retrieval selection data. The one or more memorydrives can be selected based on the available utilization levels of theset of memory drives and/or can otherwise be selected based on the driveutilization data 2425. For example, one or more memory drives withhighest levels of available utilization at a given period of time can beidentified, where this one or more memory drives with highest levels ofavailable utilization are selected for access in generating the segmentretrieval selection data at the given period of time. As anotherexample, one or more memory drives with lowest raw utilization metricsor estimates can be selected. As another example, one or more memorydrives with lowest percentages of utilization can be selected. Bycontinually selecting the least-utilized drive and/or the drive with thegreatest amount of under-utilization relative to its maximum utilizationthreshold over time, IO parallelism can be maximized because one driveisn't overscheduled above its maximum throughout threshold beforescheduling other, under-utilized drives first.

Once these one or more memory drives are selected, one or moreparticular segments can be selected for retrieval from the one or moreselected memory drives. As illustrated in FIG. 24D, the segmentprocessing assignment module can receive, access, and/or determinesegment-to-drive mapping data 2426 indicating where segments in thesegment set are stored and/or indicated a listing or lookup table of allsegments stored in each memory drive 2440-1-2440-M. Thissegment-to-drive mapping data 2426 can be utilized to determine a set ofpossible segments for selection, where the set of possible segmentscorrespond to only segments in the segment sets of the query set thatare stored in the one or more selected memory drives, and that arepending or otherwise have not yet been processed. As indicated in FIG.24D, segment set can indicate which ones of the set of segments havealready been retrieved, and which ones of the set of segments arepending or otherwise have yet to be requested for retrieval. The segmentset can further indicate which ones of the set of segments are currentlybeing retrieved, where retrieval of the segment has been initiated basedon being previously indicated in segment processing selection data 2428of a prior time slice, but where retrieval of the segment has not beencompleted by the segment processing module 2430. Alternatively, thisinformation indicating retrieval status of segments in the segment setscan be stored elsewhere and/or can be determined separately fromaccessing the query set 2415.

The final set of segments to be identified for retrieval in the giventime slice can be selected from the possible set of segments based on arandom or pseudo-random selection, based on an ordering of the segmentsindicated in the segment set, and/or based on a deterministic selection.Determining the final set of segments can include selecting a number ofsegments to be selected. For example, larger numbers of segments can beselected for retrieval from one or more drives based on the level ofunder-utilization of each of the one or more drives, where greaternumbers of segments are selected for retrieval for a time slice from amemory drive that is greatly under-utilized, and smaller numbers of areselected for retrieval for a same or different time slice from a memorydrive that is only slightly under-utilized. In other cases, the samenumber of segments, such as exactly one segment, is always selected.

In some cases, other factors are utilized to select the final set ofsegments from the possible set of segments. This can include selectingone or more of a subset of the set of queries with segments in thepossible set of segments, where the segments are deterministically,randomly, or pseudo-randomly selected from the possible set of segmentsthat are included in segment sets of the selected one or more queries.For example, a query that has the fewest remaining segments forprocessing across queries in the subset can be selected; a query in thesubset that is being processed by a particular processing core resources48 that is determined to be most under-utilized and/or that isunder-utilized with respect to a processing core resources utilizationthreshold can be selected; a query in the subset whose execution hasbeen initiated via prior retrieval of at least one different segment ofthe query's segment set can be selected over another query in the subsetwhose execution has not yet been initiated; a query in the subset with ahighest assigned priority can be selected over a query in the subsetwith a lower assigned priority; and/or other information regarding thequeries in the subset can be utilized to select one or more particularqueries from the subset of queries to have segments retrieved in thegiven time slice.

Another selection factor can include determining if the set of possiblesegments include any sets of segments that are stored sequentially in amemory drive that can be retrieved via a single request for the range ofmemory that includes the set of sequentially stored segments. In somecases, sequentially stored segments can be included in the segment setof the same query and/or of different queries in the query set. In suchcases, some or all of the identified sequentially stored segments can beselected for retrieval in a batched request to the memory drive, forexample, for retrieval and processing via the same one of the set ofprocessing core resources 48. Selecting ones of the identifiedsequentially stored segments can further include selecting thedetermined number of segments from the set of sequentially storedsegments.

Rather than automatically selecting the most under-utilized memorydrives for segment retrieval, segments from other drives determined tobe under-utilized can be selected for retrieval. In such embodiments,the available utilization levels can be compared to a predeterminedmaximum utilization availability threshold, where a proper subset ofmemory drives with available utilization levels are greater than themaximum utilization availability threshold or that otherwise compareunfavorably to the maximum utilization availability threshold isidentified. The maximum utilization availability threshold can be thesame across all memory drives regardless of whether they have the sameor different maximum utilization thresholds.

Alternatively, a set of threshold utilization levels can be determinedfor each of the set of memory drives, where each of the set of thresholdutilization levels are the same or different based on having same ordifferent corresponding maximum utilization thresholds, and/or whereeach of the set of threshold utilization levels are determined based ona predetermined difference from and/or predetermined proportion of thecorresponding set of maximum utilization thresholds. Some or allthreshold utilization levels can be strictly less than the correspondingmaximum utilization level, or can be equal to the maximum utilizationlevel. The raw and/or estimated level of utilization indicated in thedrive utilization data for each of the set of memory drives can becompared to their respective threshold utilization levels, where theproper subset of memory drives is alternatively determined byidentifying ones of the set of memory drives with utilization levelsthat are less than their respective threshold utilization level or thatotherwise compare unfavorably to the utilization availability threshold.

Once the proper subset of memory drives is identified via eithermechanism described above or by a different determination, the one ormore memory drives can be selected from this proper subset of memorydrives. For example, all of the proper subset of memory drives can beselected where at least one segment is identified for retrieval fromeach of the proper subset of memory drives. Alternatively, at least oneof the proper subset of memory drives is not selected, for example,based on determining to select a predetermined number of memory drivesthat is less than the predetermined number of memory drives and/or apredetermined number of segments that is less than the size of theproper subset of memory drives. For example, the one or more memorydrives can be selected from the proper subset of memory drives randomlyor pseudo-randomly, can be selected from the proper subset of memorydrives in accordance with a round robin scheme over time, and/or can beselected based on another determination.

In some cases, the one or more memory drives are not selected from theproper subset of memory drives, and instead one or more segments areselected from a larger set of possible segments, where this larger setof possible segments correspond to all segments in any segment set ofthe query set that are stored in any of the determined proper subset ofmemory drives. For example, rather than selecting a segment forretrieval from the most under-utilized drive, a different segment isselected from another under-utilized drive that is not necessarily themost under-utilized, based on its utilization comparing unfavorably toits threshold utilization level or its utilization availabilitycomparing unfavorably to the maximum utilization availability level.This can be ideal as other optimizations relating to the segmentsthemselves can be utilized to intelligently select particular segmentsthat are stored in any under-utilized drive for retrieval.

FIGS. 24E and 24F illustrate example embodiments of selecting differentsegments for retrieval in different time slices t₀ and t₁ respectively,where time slice t₀ occurs immediately before time slice t₁ in theplurality of sequential time slices. As illustrated in FIG. 24E, driveutilization data 2425 determined for time slice t₀ indicates utilizationlevels of 70%, 50%, and 80% for memory drives 1, 2, and M, respectively.Assume for this example that 50% is lower than utilization level acrossadditional memory drives 3-M-1. Thus, memory drive 2 is selected forsegment retrieval for time slice t₀ by the segment processing assignmentmodule 2420 because memory drive 2 has the lowest level of utilizationand/or because it has a highest amount of available utilization. Segment3 is then selected for retrieval by the segment processing assignmentmodule 2420 because it is determined to be stored in memory drive 2, andbecause it has not yet been retrieved for processing of query 2. Thisselection of segment 3 for retrieval is indicated in the segmentprocessing selection data generated for time slice t₀. Segment 3 isretrieved from memory drive 2 by segment processing module 2430 based onthe segment processing selection data 2428 generated for time slice t₀indicating selection of segment 3 for retrieval.

As illustrated in FIG. 24F, the drive utilization data 2425 determinedfor time slice t₁ has changed from the drive utilization data 2425determined for time slice t₀ illustrated in FIG. 24E. Drive utilizationdata 2425 determined for time slice t₁ indicates utilization levels of20%, 70%, and 60% for memory drives 1, 2, and M, respectively. Theincrease of utilization level for memory drive 2 can be due to retrievalof segment 3 initiated at time slice t₀ still being in progress at timeslice t₁ and/or can be due to other memory drive utilization inducedsince determining drive utilization data for time slice t₀. The decreaseof utilization for memory drive 1 can be due to a previously initiatedretrieval of other segments from memory drive 1 that were in progresswhen drive utilization data for time slice t₀ completing prior to driveutilization data determined for time slice t₁ and/or can be based onother utilization of the memory drive in this time frame between timeslice t₀ and time slice t₁. Assume for this example that 20% is lowerthan utilization level across additional memory drives 3-M-1. Thus,memory drive 1 is selected for segment retrieval for time slice t₀ bythe segment processing assignment module 2420 because memory drive 2 hasthe lowest level of utilization and/or because it has a highest amountof available utilization. Segments 2 and X are then selected forretrieval by the segment processing assignment module 2420 because itthey are determined to be stored in memory drive 2, and because theyhave not yet been retrieved for processing of queries N and 1,respectively. This selection of segment 3 for retrieval is indicated inthe segment processing selection data generated for time slice t₀.Segment 2 and segment X are retrieved from memory drive 2440-1 bysegment processing module 2430 based on the segment processing selectiondata 2428 generated for time slice t₁ indicating retrieval of segment 2and segment X.

In this case, multiple segments may have been selected for retrievalfrom memory drive 2440-1 in time slice 1 based on the level ofutilization of memory drive 2440-1 being particularly low, and/or basedon the level of utilization of memory drive 2440-1 determined for timeslice t₁ indicating higher utilization availability than the utilizationavailability determined for memory drive 2440-1 for time slice t₀ thatyielded selection of only one segment for retrieval from memory drive2440-2. For example, the number of segments selected for retrieval froma particular memory drive in a particular time slice can be anincreasing function of the memory drive's utilization availability. Insuch cases, the multiple segments from the same memory drive can beselected by selecting ones of the possible set of segments that areincluded in different query's segment sets, for example, to distributeexecution across different queries as evenly as possible. Alternativelyor in addition, different processing core resources 48 can be selectedfor retrieval of the different segments from the same memory device forexample, to ensure none of the processing core resources are overloadedwith retrieval and processing of too many segments and/or to distributeretrieval and/or processing of queries across the processing coreresources and/or parallel threads as evenly as possible. In some cases,the number of time slices retrieved in the given time slice is cappedbased on current utilization and/or resource availability of segmentprocessing module 2430 and/or of individual processing core resources48.

FIGS. 24G-24K illustrate examples of a segment scheduling module 2410that implements a utilization data generating module 2450 to generatethe drive utilization data 2425 for some or all time slices.Alternatively, the utilization data generating module 2450 can beimplemented by different a processing module of the node 37 thatcommunicates with the segment scheduling module 2410 to send the segmentscheduling module the drive utilization data 2425.

In the examples illustrated in FIGS. 24G-24K, the utilization datagenerating module 2450 generates the utilization data based on trackingthe initiation and/or completion of segment retrieval over the pluralityof time slices to determine how many segments are currently beingretrieved by the node 37 from each memory drive at any given time slice.As illustrated in FIG. 24G, the utilization data generating module 2450can generate drive utilization data 2425 for time slice t₀, and can sendthis information to the segment processing assignment module 2420. Uponreceiving this drive utilization data 2425 determined for time slice t₀is then utilized by the segment processing assignment module 2420 togenerate the segment processing selection data 2428 for use by thesegment processing module to initiate retrieval of these segments intime slice t₀ as discussed previously. Note that the utilization datagenerated for time slice t₀ as illustrated in FIG. 24G can correspond toexpected utilization for the time slice t₀ corresponding to the span oftime when the new segments indicated in segment processing selectiondata 2428 have their retrieval initiated, and/or corresponds to the mostrecent utilization data leading up to time slice t₀ when the newsegments indicated in segment processing selection data 2428 have theirretrieval initiated.

As illustrated in FIG. 24H, some or all of segment processing selectiondata 2428 for time slice t₀ can also be sent back to the utilizationdata generating module 2450. This allows the utilization data generatingmodule to determine which segments are currently in the process of beingretrieved and/or that will be in the process of being retrieved in thenext time slice t₁ of the plurality of sequential time slices, and/or inmultiple subsequent next time slices starting at t₁. Alternatively or inaddition, the segment processing selection data 2428 is utilized toupdate the status of segments in query set 2415 to indicate that theyhave their retrieval initiated at time slice t₁ or to update anotherrecord accessible by the node 37 tracking which segments in the queryset 2415 are currently in the process of being retrieved.

The utilization data generating module 2450 can determine whetherretrieval of one or more other previously requested segments selected insegment processing selection data 2428 of one or more time slices priorto time slice t₀ have completed. This can include determining whetherthe status of the segment in the query set 2415 and/or other recordindicates that their retrieval is complete. For example, the segmentprocessing module 2430 can send notifications to the segment schedulingmodule 2410 indicating completion of retrieval of segments upon theircompletion, or the segment scheduling module 2410 can otherwisedetermine when the retrieval has completed.

Alternatively or in addition, the utilization data generating module2450 can determined whether retrieval of one or more other previouslyrequested segments selected in segment processing selection data 2428 ofone or more time slices prior to time slice t₀ are expected to havecompleted, for example, if actual notifications indicating theircompletion are delayed with respect to the rate of the plurality timeslices and/or if this information is not received. The time thatretrieval of a given segment is expected to be completed can be based onan estimated retrieval time for the given segment and/or estimatednumber of time slices from the time the retrieval is initiated untilretrieval of the given segment is complete. The estimated retrieval timeor estimated number of time slices can be utilized in conjunction withthe known time slice that retrieval was initiated in correspondingsegment processing selection data 2428 to determine an expected timeslice that the retrieval of the time slice will be completed, forexample, by adding the estimated retrieval time or estimated number oftime slices to the time retrieval was initiated.

This estimate can be determined in conjunction with other segments beingconcurrently retrieved, for example, by the same processing coreresource 48. For example, the estimated amount of time to retrieve aslice can be an increasing function of the number of segments beingretrieved from the same or different memory drive by the particularprocessing core resource 48 and/or by the segment processing module 2430as a whole. This estimate can be determined in conjunction with othersegments being retrieved, for example, from the same memory drive 2440.For example, the estimated amount of time to retrieve a slice can be anincreasing function of the number of segments currently being retrievedfrom the same memory drive 2440 by the same or different processing coreresource 48 and/or by the segment processing module 2430 as a whole.

The estimated retrieval time for the given segment can be the same ordifferent for segments retrieved from different memory devices. Forexample, the estimated retrieval time can based on the memory drive,where different memory drives have different estimated retrieval timesbased on the type of memory device being utilized to implement thememory drive and/or based on historical time of retrieval of segmentsfrom different memory drives. The estimate retrieval time can be basedon the segment being retrieved, such as the size of the segment, thelocation of the segment on the memory drive, and/or the type ofencoding, encryption, compression, and/or other storage mechanismutilized to store the segment on the memory drive. Different segments ofdifferent sizes, in different locations on the same memory drive, and/orstored via different types of storage mechanisms can have differentcorresponding estimated retrieval times based on these differencesand/or based on historical retrieval times of these different types ofsegments.

The utilization data generating module 2450 can determine whether or noteach previously requested segment is known or expected to have itsretrieval completed at the time the next drive utilization data for thenext time slice is generated and/or whether or not each previouslyrequested segment is known or expected to have its retrieval completedby the time slice for which the next drive utilization data is beinggenerated. This can be utilized to determine a set of segments for eachmemory drive 2440 with retrieval in progress for the next time slice.The number of segments in each of these sets can be utilized todetermine the utilization level of the corresponding memory drive 2440,where the utilization level is an increasing function of the number ofsegments currently being retrieved from the memory drive. Alternativelyor in addition, change in the number of segments in the set from apreviously determined set for previously generated utilization data forthe memory drive can be utilized to in utilized to determine the changein utilization level from the previous utilization level, where theamount of change of utilization level is an increasing function of theamount of change in the number of segments.

For example, as illustrated in FIG. 24I, updated drive utilization data2425 is generated for time slice t₁ based on one or more previouslygenerated segment processing selection data 2428 and/or based on thenumber of segments determined or expected to be undergoing retrievalfrom each of the memory drive during the time slice t₁. This updateddrive utilization data similarly sent to the segment processingassignment module 2420. As illustrated in FIG. 24J, the segmentprocessing assignment module 2420 utilizes this updated driveutilization data to generate the segment processing selection data 2428for time slice t₁, which is sent to the segment processing module 2430and is utilized by the utilization data generating module 2450 togenerate the next updated drive utilization data. This process ofupdating the drive utilization data based on tracking which and/or howmany segments are currently being retrieved from memory drives cancontinue over time for subsequent ones of the plurality of sequentialtime slices.

In the example illustrated in FIG. 24K, the utilization data generatingmodule 2450 generates the drive utilization data based on sampling thememory drives' utilization levels in every time slice and/or in anevenly distributed proportion of time slices, where the memory drives'utilization levels are occasionally sampled in accordance with apredetermined sampling schedule and/or based on utilization metricrequests sent to the memory drives. The memory drives can generateutilization metrics by measuring or otherwise determining their currentutilization level and/or one or more recent utilization levels, such asmeasured throughput, measured processing resource utilization, and/orother information indicating a measured level of utilization of thememory drive. These one or more utilization metrics can be measured andset to the utilization data generating module 2450 in response toreceiving the request and/or in accordance with the predeterminedsampling schedule. The utilization data generating module canconsolidate, analyze and/or process the utilization metrics to generatethe drive utilization data 2425. Alternatively, another processingmodule of the node's computing device 18 can monitor and/or sampleutilization of the node's memory drives 2440 and/or all memory drives2440 of all of the plurality of nodes 37 implemented by the computingdevice 18 to generate utilization metrics for some or all of thesememory drives 2440 of the computing device 18, where this processingmodule sends utilization metrics corresponding to the particular node'smemory drives to the segment scheduler module 2410 in scheduledintervals and/or in response to requests, and/or where the segmentscheduler module 2410 otherwise accesses the utilization metricsgenerated by the processing module of the computing device 18.

This sampling of the memory drives' utilization levels can be performedalternatively or additionally to the tracking of segment retrieval overtime as illustrated in FIGS. 24G-24J to generate the utilization data,for example, where utilization data is generated based on both theretrieved metrics and the tracked segment retrieval. Alternatively,tracked segment retrieval can be utilized to estimate changes inutilization from a most recent time slice where actual utilizationmetrics were sampled, where these estimated changes are calculated basedon segment retrieval alone for one or more time slices until a latertime slice when updated utilization metrics are received from some orall memory drives, resetting the utilization data where estimatedchanges are calculated with respect to these more recently updatedutilization metrics.

Utilizing the actual utilization metrics sampled from the memory drivesto generate utilization data can be ideal as it may provide moreaccurate information, and can further account for additional accesses orutilization of these drives, for example, by other nodes in conjunctionwith recovering segments implemented as virtual segments as discussed infurther detail herein. However, as it is inefficient and/or unideal tosample utilization very frequently, combining a less frequent samplingof actual metrics with estimated changes induced by tracked segmentretrieval by the node can be ideal in maintaining occasional updates todetermine actual drive utilization, while providing sufficient estimatesof drive utilization for time slices where the drives are not sampledbased on the tracked segment retrieval.

In various embodiments, a node of a computing device has at least oneprocessor and memory that stores executable instructions that, whenexecuted by the at least one processor, cause at least one processingmodule of the node to determine a query for execution, and to determinea set of segments required to execute the query, where the set ofsegments is stored in a set of memory drives. For each of a plurality ofsequential time slices, the executable instructions, when executed bythe at least one processor, further cause at least one processing moduleof the node to determine utilization data for the set of memory drives,to select at least one of the set of memory drives based on theutilization data, and to retrieve one or more of the set of segmentsstored in the at least one of the set of memory drives to facilitate oneor more of a set of partial executions of the query utilizing the one ormore of the set of segments. Each of a plurality of selected at leastone of the set of segments are retrieved in a corresponding one of theplurality of sequential time slices, where each of the set of partialexecutions are facilitated utilizing a corresponding one of theplurality of selected at least one of the set of segments. Facilitationof the plurality of partial executions yields execution of the query.

FIGS. 24L and 24M illustrate a method for execution by a node 37. Forexample, the node can utilize at least one processing module of the node37 to execute operational instructions stored in memory accessible bythe node, where the execution of the operational instructions causes thenode 37 to execute the steps of FIG. 24L. The method of 24L can beperformed by a node 37 in accordance with embodiments of node 37discussed in conjunction with FIGS. 24A-24K, and/or in conjunction withother embodiments of node 37 discussed herein.

Step 2482 includes determining a query for execution. For example, thequery can be received by the node for execution. Step 2484 includesdetermining a set of segments required to execute the query, where theset of segments is stored in a set of memory drives. Step 2486 includes,for each of a plurality of sequential time slices: determiningutilization data for the set of memory drives; selecting one of the setof memory drives based on the utilization data; and/retrieving one ofthe set of segments stored in the one of the set of memory drives tofacilitate at least one of a set of partial executions of the queryutilizing the one of the set of segments. Each of the set of segments isretrieved in a corresponding one of the plurality of sequential timeslices. Each of the set of partial executions are facilitated utilizinga corresponding one of the set of segments. Facilitation of the set ofpartial executions yields execution of the query.

The three steps of step 2486 that are be performed for each of theplurality of sequential time slices are illustrated as a method in FIG.24M, where the method of FIG. 24M is repeated for each of the each ofthe plurality of sequential time slices to render execution of step 2486of FIG. 24L. Step 2488 includes determining utilization data for a setof memory drives. Step 2490 includes selecting one of the set of memorydrives based on the utilization data. Step 2492 includes retrieving oneof a set of segments stored in the one of the set of memory drives tofacilitate at least one of a set of partial executions of a queryutilizing the one of the set of segments.

In various embodiments, determining the utilization data includesdetermining a plurality of utilization levels. Each of the plurality ofutilization levels corresponds to one of the set of memory drives, andthe one of the set of memory drives is selected based on the at one ofthe set of memory drives having a most unfavorable utilization level ofthe plurality of utilization levels. In various embodiments, each of theplurality of utilization levels are determined based on determiningcurrent resource utilization metrics for each of the set of memorydrives by sampling each of the set of memory drives. In variousembodiments, for one of the plurality of sequential time slices, a firstutilization level of the plurality of utilization levels is determinedfor a first one of the set of memory drives, and a second utilizationlevel of the plurality of utilization levels is determined for a secondone of the set of memory drives. The first utilization level is moreunfavorable than the second utilization level based on the first one ofthe set of memory drives having first current resource utilizationmetrics indicating lower resource utilization than second currentresource utilization metrics of the second one of the set of memorydrives. The first one of the set of memory drives can be selected forthe one of the plurality of sequential time slices in response to havingthe most unfavorable utilization level of all of the utilization levelsfor the one of the plurality of sequential time slices.

In various embodiments, the plurality of utilization levels aredetermined based on determining at least one prior subset of the set ofsegments retrieved in at least one corresponding prior time slice of theplurality of sequential time slices. In various embodiments, for one ofthe plurality of sequential time slices, retrieval of a first priorsubset of the set of segments from a first one of the set of memorydrives was initiated within a subset of prior time slices of theplurality of sequential time slices, and retrieval of a second priorsubset of the set of segments from a second one of the set of memorydrives was initiated within the subset of prior time slices of theplurality of sequential time slices. A first utilization level of theplurality of utilization levels is determined for the first one of theset of memory drives for the of the plurality of sequential time slicesbased on a first number of segments in the first prior subset of the setof segments. A second utilization level of the plurality of utilizationlevels is determined for the second one of the set of memory drives forthe of the plurality of sequential time slices based on a second numberof segments in the second prior subset of the set of segments. The firstutilization level is more unfavorable than the second utilization levelbased on the first number of segments in the first prior subset of theset of segments being lower than the second number of segments in thesecond prior subset of the set of segments. The first one of the set ofmemory drives can be selected for the one of the plurality of sequentialtime slices in response to having the most unfavorable utilization levelof all of the utilization levels for the one of the plurality ofsequential time slices. In various embodiments, the method furtherincludes determining, for the one of the plurality of sequential timeslices, the first prior subset of the set of segments and the secondprior subset of the set of segments based on determining ones of the setof segments whose retrieval is currently in progress during the one ofthe plurality of sequential time slices.

In various embodiments, one of the set of memory drives is determined tohave a most unfavorable utilization level of the plurality ofutilization levels based on having a utilization level indicating alowest level of current resource utilization of the plurality ofutilization levels. In various embodiments, the method further includesdetermining a maximum throughput for each of the set of memory drives,and determining available utilization for each of the set of memorydrives based on a difference between the maximum throughput of the eachof the set of memory drives and the utilization level of the each of theset of memory drives. One of the set of memory drives is determined tohave a most unfavorable utilization level of the plurality ofutilization levels based on having a highest available utilization ofthe set of memory drives.

In various embodiments, the method includes determining a plurality ofqueries for execution that includes the query. The method furtherincludes determining a plurality of sets of segments by determining, foreach of the plurality of queries, a corresponding set of segmentsrequired to execute the query, where the plurality of sets of segmentsis stored in the set of memory drives. One of the plurality of sets ofsegments is retrieved for each of the plurality of sequential timeslices based on the selection of the one of the set of memory drivesbased on the utilization data. Each partial execution of a plurality ofsets of partial executions are facilitated utilizing a corresponding oneof the plurality of sets of segments, and facilitation of each set ofpartial executions in the plurality of sets of partial executions yieldsexecution of a corresponding one of the plurality of queries.

In various embodiments, a first time slice of the plurality ofsequential time slices includes a retrieval of a first one of theplurality of sets of segments. A second time slice of the plurality ofsequential time slices includes a retrieval of a second one of theplurality of sets of segments. A third time slice of the plurality ofsequential time slices includes a retrieval of a third one of theplurality of sets of segments. The first time slice is before the secondtime slice in the plurality of sequential time slices, and the secondtime slice is before the third time slice in the plurality of sequentialtime slices. The first one of the plurality of sets of segments isutilized to facilitate a partial execution of a first one of theplurality of queries. The second one of the plurality of sets ofsegments is utilized to facilitate a partial execution of a second oneof the plurality of queries. The third one of the plurality of sets ofsegments is also utilized to facilitate a partial execution of the firstone of the plurality of queries.

In various embodiments, a non-transitory computer readable storagemedium includes at least one memory section that stores operationalinstructions that, when executed by a processing module that includes aprocessor and a memory, cause the processing module to determine a queryfor execution, and to determine a set of segments required to executethe query, where the set of segments is stored in a set of memorydrives. For each of a plurality of sequential time slices, theexecutable instructions, when executed by the at least one processor,further cause at least one processing module to determine utilizationdata for the set of memory drives, to select at least one of the set ofmemory drives based on the utilization data, and to retrieve one or moreof the set of segments stored in the at least one of the set of memorydrives to facilitate one or more of a set of partial executions of thequery utilizing the one or more of the set of segments. Each of aplurality of selected at least one of the set of segments are retrievedin a corresponding one of the plurality of sequential time slices, whereeach of the set of partial executions are facilitated utilizing acorresponding one of the plurality of selected at least one of the setof segments. Facilitation of the plurality of partial executions yieldsexecution of the query.

FIGS. 25A-25I present embodiments of a node 37 that facilitate executionof queries 2405 in the query set 2415, even when at least one segment2445 indicated in the corresponding segment set 2418 is unavailable foraccess in segment storage 2442. The embodiments illustrated in 25A-25Ican be utilized to implement some or all of the plurality of nodes 37 ofsome or all computing devices 18-1-18-n, for example, of the of theparallelized data store, retrieve, and/or process sub-system 12, and/orof the parallelized query and results sub-system 13. The embodiments ofnode 37 discussed in conjunction with FIGS. 25A-25I can be utilized toimplement any other nodes 37 of database system 10 discussed herein.

Segments on disk may become unavailable due to drive failures, such as afailure of one or more corresponding memory drives 2440 responsiblestoring these segments. Segment unavailability can also occur as aresult of failure of a corresponding node 37 storing the segment;software issues such as software issues regarding the node 37 and/ormemory drive 2440 storing the segment; connectivity problems such asconnectivity problems that render retrieval of segments from theirmemory drive 2440 and/or node 37 unavailable; and/or other issues thatresult in unavailability of access to a segment via its memory drive2440 at a given time.

In embodiments where segments are not replicated or otherwise do nothave duplicate segments stored on other memory drives of the same ordifferent node 37 and/or computing device 18, unavailability of aparticular segment can render a corresponding query requiring thesegment as impossible to execute properly, as the query resultant is notguaranteed to be correct if at least one segment in the query's segmentset is not processed. However, in embodiments of database system 10discussed herein, segments are stored in exactly one location and are“owned” by exactly one node.

As discussed previously in conjunction with FIGS. 15-23 , segments canbe generated in accordance with a redundancy storage encoding scheme.Thus, despite unavailability of access to a particular segment via itscorresponding memory drive 2440, this segment can be reconstructed inaccordance with the redundancy storage encoding scheme, so long asenough other segments in the same segment group as the segment, asdictated by the redundancy storage encoding scheme, are available foraccess on their respective memory drives and/or so long as segmentlosses do not otherwise exceed the failure threshold of the redundancystorage encoding scheme.

When a segment becomes unavailable, processing resources of databasesystem 10 can fully rebuild these segments. As used herein, fullrebuilding refers to the process of reconstructing at least oneunavailable segment in accordance with the corresponding redundancystorage encoding scheme utilized to generate the segment, and furtherre-homing these reconstructed segments for storage in the same ordifferent memory drive 2440 and/or for storage by segment storage 2442of the same or different node 37 and/or the same or different computingdevice 18. In particular, if a particular memory drive 2440 fails, forexample, due to failure of a particular memory device 42 that implementssome or all of the memory drives 2440, the plurality of segments storedby this memory drive 2440 can be rebuild by the database system 10, forexample, for storage on a new memory drive 2440. This can includeassigning these rebuilt segments to a new node in response to rebuildingthe segments on one or more memory drives 2440 of a new node 37, whererebuild segments of the memory drive 2440 can be assigned to storage viathe same or different new node 37. Alternatively, a replacement memorydrive 2440 or repaired original memory drive 2440 can be implemented onthe original node, where the rebuilt segments are stored on thereplacement or repaired memory drive and are assigned to storage via thesame node 37.

However, this full rebuilding process can be lengthy, particularly if alarge number of segments from a failed memory drive need to bereconstructed and then re-stored on disk in accordance with the fullrebuilding process. Relying on completion of this full rebuildingprocess for access to unavailable segments to facilitate execution ofongoing queries can therefore be unideal, as these queries cannot beperformed or completed during the entire, lengthy duration of this fullrebuilding process.

Furthermore, as the rebuilding process of all segments of an unavailablememory drive and/or entire node is lengthy and computationallyexpensive, there are cases where the database system 10 may wish toforego the rebuilding process. In particular, if a memory drive or nodeoutage is temporary, for example, due to scheduled and/or otherwiseknown maintenance and/or due to a scheduled migration of segments to anew memory drive of the same or different node on a same or differentcomputing device 18, rebuilding of all segments of these temporarilyunavailable memory drives or entire nodes is unideal, as access to thesegments will again become available once the temporary outage haselapsed, and expensive rebuilding is not necessary. However, these casesof waived rebuilding also pose problems for timely query execution, asqueries requiring nodes stored on memory drives or nodes undergoing thetemporary outage cannot be performed or completed during this temporaryoutage.

The embodiments discussed in conjunction with FIGS. 25A-25L addressthese problems by enabling dynamic recovery of unavailable segmentsrequired for execution of ongoing queries without requiring theseunavailable segments undergo the full rebuilding process of being fullyrebuilt to disk. This can be ideal, as segments that are not requiredfor query during a scheduled outage are not recovered, where only theproper subset of unavailable segments during the outage that arerequired for query execution during the outage are recovered.Furthermore, individual segments that may be undergoing a fullrebuilding process to be rebuild to disk by the database system 10 aspart of a large rebuilding operation to rebuild a large number ofsegments can also be dynamically recovered for execution of queries,separately from this rebuilding process and/or before this fullrebuilding process is completed. This ability to dynamically recoverunavailable segments as they are required for execution of ongoingqueries improves database systems by enabling queries to be executedduring outages and/or before unavailable segments are fully rebuilt todisk, ensuring that pending queries do not undergo long delays due tooutages and/or lengthy rebuilding operations, while preserving resourcesto only recover the unavailable segments as they are necessary for queryexecution.

FIG. 25A illustrates an example of a node's execution of a query thatrequires retrieval and/or processing of at least one unavailablesegment. As discussed thus far with regards to the embodiments of24A-24L, a node retrieves segments 2445 from segment storage 2442 toservice incoming or ongoing queries of query set 2415. These segmentsthat are stored in segment storage 2442, are otherwise stored to disk,are currently available for retrieval from their storage withoututilizing a recovery scheme, and/or that are designated for retrievalfrom their storage without utilizing a recovery scheme can correspond to“physical segments.” When physical segments become unavailable, a node37 can be implemented to perform dynamic recovery of the unavailablesegments that are required for execution of a query in query set 2415 byimplementing a segment recovery module 2510. The segment recovery module2510 can be implemented utilizing at least one processing module of thenode 37, such as at least one processing core resource 48. Theunavailable segments that a node determined to recover to service itsexecution of at least one query can correspond to “virtual segments”that are reconstructed and/or otherwise recovered to service the query.

As used herein, a “virtual segment” refers to the abstraction and/orlogical construct of a locally rebuilt segment that is not permanentlyre-homed and/or is not stored to disk. A segment being processed as avirtual segment can require generation of a “locally rebuilt segment” or“recovered segment” via one or more other segments that are physicallystored elsewhere to service ongoing and/or incoming queries. Virtualsegments can correspond to segments determined to be unavailable intheir designated memory drive 2440 of segment storage 2442 of the sameor different node, that therefore must be recovered if needed to servicequeries while unavailable physically. In some cases, such as theembodiments discussed in conjunction with FIGS. 27A-27K, a segmentprocessed as a virtual segment may be currently available for access asa physical segment by the same or different node, but is determined tobe processed as a virtual segment by generating a recovered segmentanyways, for example, based on a node segment set of data ownershipinformation indicating that the segment be processed by the node as avirtual segment.

While not depicted in FIG. 25A, the segment recovery module 2510 can beimplemented utilizing the segment processing module 2430, can beimplemented utilizing the same set of processing resources as thesegment processing module 2430, can be implemented utilizing overlappingprocessing resources shared with the segment processing module 2430,and/or can be implemented utilizing a proper subset of the node'sprocessing resources that are allocated to the segment recovery module2510, where another, distinct proper subset of the node's processingresources are allocated to the segment processing module 2430.Embodiments enabling intelligent allocation of processing resources forutilization by the segment recovery module 2510 to recover and/orprocess unavailable segments and for utilization by the segmentprocessing module for normal retrieval and/or processing of availablesegments is discussed in further detail in conjunction with FIGS.26A-26C.

The segment recovery module 2510 can determine to recover an unavailablesegment and/or a segment otherwise indicated as a virtual segment. Forexample, as illustrated in FIG. 25A, the segment recovery module canreceive a request from the segment processing module 2430 indicating thesegment to be recovered. Alternatively or in addition, this informationcan be received from the segment scheduling module 2410 and/or can bebased on segment processing selection data 2428 that identifies that theunavailable segment be retrieved and/or processed in conjunction with apartial execution of a query in the query set 2415. Alternatively or inaddition, the segment recovery module 2510 can otherwise determine theto recover the unavailable segment based on determining the unavailablesegment is required for execution of an ongoing query and/or a new queryin the query set 2415.

In response to determining to recover an unavailable segment and/orvirtual segment, the segment recovery module 2510 can facilitaterecovery of the segment to generate a locally rebuilt segment byimplementing a recovery scheme. This can include determine a set ofother segments 1-K that are stored on disk by other nodes, and that arerequired to recover the unavailable segment. This can includedetermining the other segments 1-K based on identifying the segmentgrouping to which the unavailable segment belongs and/or to otherwisedetermine the particular segments needed to recover the unavailablesegment based on accessing metadata structures associated with thesegment, a lookup table, accessing information storing indicating thesegment groupings, or otherwise determining the segment grouping basedon an identifier of the unavailable segment. This can also includeselecting the other segments 1-K from a larger set of possible segmentsthat will be recovered, where any K segments of the larger set ofpossible segments can be utilized to recover the unavailable segment.Alternatively, requests for all of the set of possible segments can besent to a larger set of other nodes, where K segments, such as the firstK received segments, are selected from a larger set of incoming possiblesegments for recovery of the unavailable segment.

Once the set of other segments 1-K are identified, the node can queryother nodes 37, such as nodes on a set of other computing devices 18,for the segments 1-K. For example, as illustrated in FIG. 25A, thesegment recovery module 2510 can generate a plurality of externalretrieval requests 1-K that are sent by the node 37 to a set of othernodes 37-1-37-K of the database system 10. Each one of the set of othernodes 37-1-37-K can be stored on a distinct one of a set ofcorresponding computing devices 18-1-18K. Each external retrievalrequest 1-K can identify the corresponding segment in the set ofsegments 1-K to be retrieved from the corresponding other node. The node37 can send the set of external retrieval requests 1-K to the set ofother nodes 37-1-37-K by utilizing the network connection 41 of the node37 and/or by utilizing system communication resources 14 of the databasesystem 10. Each other node 37-1-37-K can receive the corresponding oneof the set of external retrieval requests 1-K via their own networkconnection 41 and/or via another connection to system communicationresources 14.

Note that, as illustrated in FIG. 25A and/or in further figures herein,nodes labeled 37-1, 37-2, . . . can correspond to different particularexample nodes for each different example discussed, for example, wherenodes are implemented by the same or different set of computing devices18 in each example. In particular, in this example and in furtherexamples herein, the example nodes 37-1, 37-2, 37-3, and 37-4 can bedifferent from example nodes 37-1, 37-2, 37-3, and 37-4 of FIGS. 7 8,and/or 9. In particular, the set of nodes 37-1, 37-2, 37-3, and/or 37-4discussed in conjunction with FIG. 25A and/or further figures herein canall be included on more than one computing device 18 and/or can each beimplemented by a different computing device.

In response to receiving an external retrieval request indicating aparticular segment for retrieval, each of the other nodes 37, and/orother processing resources of each computing device 18 that includes acorresponding one of the other nodes, can facilitate retrieval of therequested segment from their own segment storage 2442. This can includeeach of the other nodes identifying which of their set of memory drives2440 of their corresponding segment storage 2442 stores the requestedsegment, retrieving the segment from the identified one of the set ofmemory drives 2440, and sending the segment back to the requesting node37. Each other node 37 can send the corresponding segment retrieved fromtheir segment storage 2442 to the requesting node 37 by again utilizingtheir network connection 41 and/or by utilizing system communicationresources 14 of the database system 10, where the requesting node 37receives the set of segments 1-K from the set of other nodes 1-K via itsnetwork connection 41 and/or via another connection to systemcommunication resources 14. In some embodiments, a segment schedulingmodule 2410 of one of these nodes 1-K can facilitate partial executionof a query requiring the requested segment in response to retrieving therequested segment from its memory device 2440. For example, rather thanfacilitating two different retrievals of this segment for the recoveryand for its own use, this particular node can perform a single retrievalof the segment to service both the request from the other node for thesegment as part of a recovery of a virtual segment being performed bythe other node, as well as this particular node's own usage of thesegment as a physical segment to generate a partial resultant for aquery 2405 with this segment in its segment set 2418.

In other embodiments, the requesting node 37 may not have knowledge ofwhich particular nodes in the database system 10 store the necessarysegments 1-K. In these cases, the node 37 can instead broadcast theexternal retrieval request to some or all nodes in the database system10 via its network connection 41 and/or via another connection to systemcommunication resources 14, where the external retrieval requestindicates the set of segments 1-K. Each other node in the databasesystem 10, upon receiving the broadcast, can determine whether theystore any of the indicated segments 1-K, where many of the nodes willdetermine that they do not store any of these segments and will thus donothing. Only the particular set of other nodes 37-1-37-K do each storeone of the requested segments will retrieve the one requested segmentfrom their segment storage 2442 accordingly for transmission back to therequesting node 37.

Once the set of segments 1-K are received by the requesting node 37, thesegment recovery module 2510 can further implement the recovery schemeto facilitate recovery of the unavailable segment. This can includegenerating a recovered segment as a locally rebuilt segment by utilizingthe set of segments 1-K. This can include utilizing the redundancystorage encoding scheme, such as an erasure scheme or other rebuildingscheme, to recover the segment. For example, a corresponding redundancystorage decoding function can be performed on the set of segments 1-K togenerate the recovered segment in accordance with the particularredundancy storage encoding scheme utilized to generate the segmentgroup that includes the unavailable segment and the set of segments 1-K.

The segment processing module 2430 can process the recovered segmentgenerated by locally rebuilding segment recovery module 2510 as avirtual segment to service the given query, and can further processother physical segments to service the given query, for example, viarequests to retrieve these physical segments from memory drives 2440 ofsegment storage 2442 as discussed previously. At least one partialresultant can be generated by utilizing the virtual segment recoveredvia segment recovery module 2510 and/or any other virtual segmentsrecovered via segment recovery module 2510 for the query that are notillustrated. Other partial resultants can be generated utilizing thephysical segments retrieved from memory. The partial resultantsgenerated via processing all segments of the given query, whetherphysical or virtual, can be combined and/or processed to yield the finalquery resultant for the query.

As the recovered segment is generated for use in the corresponding queryexecuted by the node 37, the recovered segment can persist only for thelife of the query, where the recovered segment is not stored is notstored in segment storage 2442 or otherwise is not stored to disk. Forexample, the particular data of the recovered segment required toexecute the query, such as some or all of the set of rows included asdata of this segment, can be forwarded to the segment processing module2430 for use in processing the query in accordance with thecorresponding operator data 2416 of the query to generate a partialresultant of the query. The recovered segment can otherwise exist as alogical construct or abstraction rather than a physically stored segmenton disk, for example, persisting only in cache memory such as cachememory 45 of one or more particular processing core resources that areutilized by the segment recovery module 2510 to reconstruct the segmentand/or that are utilized by the segment scheduling module 2410 toprocess the recovered segment to generate the corresponding partialresultant for the query. Once the query is fully executed, the recoveredsegment can be deleted from cache memory, can be no longer available tothe node 37, and/or can otherwise no longer persist. In some cases, oncethe particular partial resultant corresponding to the recovered segmentis generated, the is recovered segment deleted from cache memory, can beno longer available to the node 37 for use in processing other queriesrequiring the virtual segment, and/or can otherwise no longer persist,even if the full query resultant has not yet been generated and/or ifthe query's execution by the node 37 is not yet complete.

Thus, because the recovered segment does not persist beyond the life ofthe query and/or beyond its use in generating the corresponding partialresult, if the particular unavailable segment is later required by thenode, for example, to service a different query, the segment recoverymodule 2510 will again re-perform the same recovery process byre-retrieving segments 1-K and re-generating the recovered segment forprocessing by the segment processing module 2430 to generate thecorresponding partial resultant in accordance with the operator data ofthe different query. While the recovered segment is interchangeablyreferred to as a “locally rebuilt segment” herein, a locally rebuiltsegment is similarly not fully rebuilt for storage on disk or in segmentstorage 2442, unlike fully rebuilt segments that are rebuilt via adifferent, full rebuilding process to fully reconstruct unavailablesegments as physical segments that are stored to disk and/or segmentstorage 2442 for subsequent retrieval as physical segment.

FIG. 25B illustrates a particular example of node 37 that processes aset of queries in query set 2415 that require use of unavailablesegments via a segment scheduling module 2410 as described inconjunction with FIGS. 24A-24L. The example of FIG. 25B illustrates thesame example storage configuration of segments in memory drives2440-1-2440-M as illustrated in FIG. 24D, and further illustrates thesame example segment sets for queries 1-N in query set 2415. However, asillustrated in FIG. 25B, the memory drive 2440-2 is experiencing a driveoutage and/or its segments are otherwise unavailable for access asphysical segments. This outage can correspond to a temporary outageand/or scheduled outage, where memory drive 2440-2 will be againavailable at a later time and where the segments of memory drive 2440-2are not rebuilt for physical storage. Alternatively, this outage cancorrespond to a permanent outage and/or unexpected outage, where thesegments of memory drive 2440-2 are rebuilt to physical storage. Forexample, within the temporal period depicted in FIG. 25B, the segmentsof memory drive 2440-2 can be currently undergoing a full rebuildingprocess for physical storage in another memory drive of the same ordifferent node 37 via other processing resources of the database system10. This current full rebuilding process can be separate from the node'srecovery of virtual segments of memory drive 2440-2 as depicted anddescribed herein, and/or this current full rebuilding process isotherwise not depicted and/or not relevant to the query execution ofFIG. 25B as described herein.

The query set 2415 can further indicate whether segments in each segmentset 2418 are to service the query as physical segments retrieveddirectly from segment storage 2442 or as virtual segment recovered viaretrieval of a set of other segments as illustrated in FIG. 25A.Alternatively, the segment scheduling module 2410 can otherwisedetermine whether each segment in segment set 2418 be processed as aphysical or virtual segment. In this example, based on the drive outageof memory drive 2440-2, the segment scheduling module 2410 can determineto process the segments stored by memory drive 2440-2 as virtualsegments. In particular, segment 3 will be processed as a virtualsegment to service queries 1 and 2, and segment Y will be processed as avirtual segment to service query 2. Some or all other segments stored bymemory drive 2440-2, such as segment 4, may not be included in any ofthe segment sets of query set 2415 during the outage of memory drive2440-2, and thus will not need to be recovered via segment recoverymodule 2510 because they are not necessary to service any queries duringthe outage.

At the time slice illustrated in FIG. 25B, the segment processingassignment module generates segment processing selection data 2428indicating that segment 3 be processed as a virtual segment for query 2,based on determining that segment 3 is unavailable for access as aphysical segment and/or based on determining that segment 3 is indicatedto be processed as a virtual segment for query 2. Segment 3 can berecovered via the segment recovery module 2510 based on the segmentprocessing selection data 2428 indicating processing of segment 3 as avirtual segment, where the recovered segment 3 generated by the segmentrecovery module 2510 is sent to and/or otherwise utilized by segmentprocessing module 2430 to generate the corresponding partial resultantfor query 2.

In this example, segment 3 will again be recovered in a later time sliceby segment recovery module 2510 to service query 1 because segment 3does not persist beyond the life of query 2 and/or beyond generating thecorresponding partial resultant for query 2. In other cases as discussedin further detail herein, recovered segment 3 can be utilized forconcurrent processing via segment processing module 2430 to generatecorresponding partial resultants for both queries 1 and 2 and/or can beutilized to service both queries 1 and 2, where segment 3 does not needto be separately recovered two distinct times by segment recovery module2510.

In the discussion of processing virtual segments described thus far,segments processed as virtual segments are recovered in their entirety.However, in other embodiments, a node 37 can further be operable todetermine that only a portion of a particular segment processed as avirtual segment is required for execution of a particular query. Forexample, a particular segment can store data corresponding to aplurality of rows of one or more particular tables, but a particularquery may only require accessing a small percentage of these rows. Asanother example, a particular segment can store a plurality of fieldsand/or columns for each of a set of rows in a particular table, but aparticular query may only require accessing a small percentage of thefields for some or all of the set of rows of the table. In such cases,rather than facilitating recovery of the entirety of a segment processedas a virtual segment, a node can instead facilitate recovery of only aportion of the segment deemed necessary to execute the correspondingquery.

For example, as illustrated in FIG. 25C, a segment 2445 can include aset of data blocks 2540 that each include a distinct set of raw rowsand/or raw columns of a table and/or portion of a table stored by thedata segment. The set of data blocks can be implemented via the codingblocks and/or data slabs discussed in conjunction with FIGS. 21 and 22 .For example, one or more particular data slabs of a segment as discussedin conjunction with FIGS. 21 and 22 can be recovered rather thanrecovering the entire segment based on determining that only the columnsincluded in the one or more particular data slabs are necessary forexecution of the query. As a particular example, if each of a pluralityof coding blocks of a segment store one or more corresponding data slabsof the segment, individual coding blocks can be recovered rather thanrecovering the entire segment. If these coding blocks are, for example,each 4 Kilo-Bytes in size as discussed in conjunction with FIG. 22 , andif each segment is, for example, 32 Giga-Bytes in size as discussed inconjunction with FIG. 22 , recovery and processing of a small number ofcoding blocks rather than an entire segment can further reducecomputation time and processing resources to generate the same requiredpartial resultant for the query. The data blocks of FIG. 25C canalternatively correspond to other distinct portions of raw data includedin a segment.

Each data block 2540 can be independently recoverable, where a singledata block 2540 can be recovered without requiring recovery of any otherdata blocks. An example of such an embodiment is illustrated in FIG.25C. A segment group includes a number of segments 2445 such as fivesegments 2445-1-2445-5 generated via the redundancy storage encodingscheme, for example, as discussed in conjunction with FIG. 23 . In thisexample, any four of the five segments can similarly be processed via adecoding scheme in accordance with the redundancy storage encodingscheme to recover the remaining one of the five segments, for example,by utilizing parity data stored in each segment generated in accordancewith the redundancy storage encoding scheme. In some embodiments, thedata blocks 2540 of each segment 2445 illustrated in FIG. 25C can befully included in the data and parity section of the correspondingsegment as illustrated in FIG. 23 , where the segments 2445 of FIG. 25Cadditionally include the manifest section, index sections, and/orstatistics section as illustrated in FIG. 23 .

In some embodiments, an individual data block of one of the fivesegments can be independently recoverable without requiring recovery ofthe entire segment. In some cases, recovery of a particular data blockof a particular segment can include retrieving a corresponding datablock from each other segment in the segment group. For example, asillustrated in FIG. 25C, each segment can be generated to include a samenumber of data blocks, where a data block in one segment belongs to adata block grouping of five data blocks that includes one data blockfrom each segment. For example, a first data block grouping can includedata blocks 2540-1-1, 2540-2-1, 2540-3-1, 2540-4-1, and 2540-5-1; asecond data block grouping can include data blocks 2540-1-2, 2540-2-2,2540-3-2, 2540-4-2, and 2540-5-2; and so on, where a total of J datablock groupings are included in the segment grouping. In particular,each particular data block can include and/or otherwise be associatedwith its own corresponding parity data that was generated in accordancewith the redundancy storage encoding scheme, where the recovery of aparticular data block of a particular segment can include utilizing theparity data specific to the corresponding block of each other segment inthe same segment group.

As a particular example, the redundancy storage encoding scheme can beindividually applied to the raw data, such as full or partial dataslabs, to be included in each data block of a data block group togenerate the data block group with its own corresponding parity datathat enables the data blocks in the data block group to be independentlyrecoverable. In this example, the redundancy storage encoding scheme canbe applied separately on J sets of data slabs or other raw data togenerate J data block groups of five data blocks each, where the dataand parity section of each segment is generated from J data blocks ofeach one of the J data block groups, and associated parity data for eachdata block if not integrated in the data blocks themselves.Alternatively, the redundancy storage encoding scheme, when applied toall raw data of each data segment to generate the segments in thesegment group as a whole, can inherently render individual portions ofthe data in each segment independently recoverable via correspondingindividual portions of data in a set of other segments of the segmentgroup and/or via the entirety of the other segments of the segmentgroup.

FIG. 25D illustrates an example where a particular data block of avirtual segment is recovered by a node 37 to process the virtual segmentto service the query. First, the segment scheduling module 2410 and/orother processing module of the node can identify that only theparticular data block is required to service the corresponding query,for example, based on domain data of the query, particular columnsaccessed in the query, and/or particular rows accessed in the query. Inthis example, data block 2540-2-2 of virtual segment 2445-2 isdetermined to be required to process query 2. This can be indicated insegment processing selection data 2428 and/or can otherwise bedetermined by the segment recovery module 2510, where the segmentrecovery module 2510 retrieves the corresponding data blocks of the setof segments 1-K in the segment group via a set of requests 1-Kindicating these corresponding data blocks. In this example, K is equalto four, and four external retrieval requests are sent to retrieve eachof four data blocks in the data block group: data blocks 2540-1-2,2540-3-2, 2540-4-2, and 2540-5-2. The retrieved data blocks areprocessed by the segment recovery module 2510 to recover the requireddata block of the virtual segment based on the redundancy storageencoding scheme utilized to encode the data blocks individually and/orutilized to encode the corresponding segments as a whole. In thisexample, the received four data blocks 2540-1-2, 2540-3-2, 2540-4-2, and2540-5-2 are processed to generate recovered data block 2540-2-2. Therecovered data block 2540-2-2 is then forwarded to segment processingmodule 2430, where the segment processing module generates thecorresponding partial resultant for the corresponding virtual segment.For example, the operator data 2416 for the corresponding query can bereceived and/or determined by the segment processing module 2430 togenerate the partial resultant. In some cases, the partial resultant ismerely the raw data, such as raw rows and/or columns of the particulardata block to be forwarded to a different node for further processing.

While not discussed in conjunction with FIGS. 24A-24L, the retrievaland/or processing of physical segments of a segment set 2418 of a query2405 can similarly include determining one or more particular datablocks 2540, such as one or more particular data slabs and/or codingblocks of FIGS. 21 and 22 , that are required to service thecorresponding query, where only a subset of data blocks identified fromthe plurality if data blocks of the segment are retrieved from thememory drive storing the segment and/or where only where only the subsetof data blocks identified from the plurality if data blocks of thesegment are processed to generate the corresponding partial result.Identifying the location of the particular data blocks 2540 of aparticular segment that will be retrieved from physical storage and/orprocessed can similarly include accessing metadata structures associatedwith the segment and/or a lookup table that indicates the location ofthe particular data blocks and/or indicates the rows and/or columnsassociated with the particular data blocks.

In some cases, the segment set 2418 includes and/or indicatesidentifiers of a set of particular data blocks 2540 of each segment2445, whether indicated to be processed as a virtual or physicalsegment, that are necessary to service the query, where these particulardata blocks, and perhaps only these particular data blocks, areretrieved from memory to service the query via a corresponding physicalsegment and/or are recovered via externally retrieved data blocks and/orexternally retrieved entire segments.

In some cases, multiple consecutive or non-consecutive data blocks ofthe segment are determined to be required to service the query, and themultiple consecutive or non-consecutive data blocks of a virtual segmentcan be recovered by independently and/or jointly recovering the multipleconsecutive or non-consecutive data blocks via corresponding multipleconsecutive or non-consecutive data blocks of each other requiredsegment in the set of segments 1-K. Identifying the multiple data blockscan include identifying a set with the minimum number of data blocksrequired to service the corresponding query, where any data blocks thatdo not include any data required to service the query are notidentified. Alternatively, in some cases, a consecutive set of datablocks is identified to simplify the transaction as a single read, evenif the consecutive set of data blocks includes one or more data blocksthat are not required to service the query.

In some cases, particularly if the required data blocks arenon-consecutive, separate data blocks of the same segment can beindicated for retrieval or recovery as separate transactions, forexample, occurring in different slices and/or scheduled by the segmentscheduling module 2410 to be performed at different times.Alternatively, multiple data blocks can be retrieved or recovered in asingle, batched request, particularly if they are consecutively storedin physical memory and/or have corresponding data blocks in the samedata block groups that are consecutively stored on other memory drivesof other nodes. In particular, the segment scheduling module 2410 canexploit the fact that many reads of data blocks and/or entire segmentsare sequential in nature. When this occurs, these reads can be batchedinto a single request. The batched reads can correspond to retrieval oflocal sequential data, such as multiple data blocks or other sequentialportions of data within the same segment and/or across multiplesegments. The batched reads can also correspond to the other segmentsand/or individual data blocks on other nodes required to be retrieved torecover a segment processed as a virtual segment. In either case, thesegment scheduling module can determine to batch required reads toretrieve multiple segments and/or multiple data blocks of the samesegment in its own memory drive or memory drives of other nodes inrecovery of a corresponding virtual segment, for example, when thisrequired data is determined to be stored sequentially and/ornon-sequentially in the same location. This can minimize the number ofdistinct requests to local memory and perhaps more importantly, canminimize the number of distinct requests sent over a network of thedatabase system 10 via system communication resources 14.

The segment scheduling module 2410 can determine multiple data blocks ofa same physical or virtual segment be retrieved and/or recovered intandem as a batched request, for example, in same segment processingselection data generated for a particular time slice and/or indicating asame core processing resource to perform the corresponding retrievaland/or recovery of the multiple data blocks. This can still includeretrieving and/or recovering only a proper subset of data blocksdetermined to be required for execution of the corresponding query. Thesegment scheduler can determine whether or not to batch requests torecover multiple data blocks of a virtual segment and/or can determinewhether or not to batch requests to recover multiple data blocks of avirtual segment. Alternatively, the segment scheduler can alwaysdetermine to batch requests to recover multiple data blocks of a samevirtual segment of a same query and/or retrieve multiple data blocks ofa same physical segment of a same query.

Similarly, batched requests to process multiple physical segments and/ormultiple virtual segments of the same query can similarly be generatedand/or determined by the segment scheduling module 2410. In particular,multiple physical segments of a segment stored in the same memory drivecan be retrieved in a single request to service a query, where themultiple retrieved segments are retrieved and processed in tandem forthe query, for example, by jointly processing the multiple retrievedsegments in accordance with the operator data to generate a singlepartial result. As another example, multiple virtual segments of thesame query can be retrieved in a batched requests, for example, if themultiple virtual segments recoverable via retrieval of segments storedon the same set of nodes 1-K, where the set of nodes 1-K are sent abatched request for retrieval of a set of multiple segments, where theset of multiple segments retrieved from each of the set of nodesconstitute at least a minimum number of required segments for each of acorresponding set of segment groups that are utilized to are to generatea set of recovered segments. The multiple recovered segments cansimilarly be jointly in accordance with the operator data to generate asingle partial result for the multiple virtual segments.

The segment scheduling module 2410 can thus similarly determine thatmultiple physical or virtual segments be retrieved and/or recovered intandem as a batched request, for example, in same segment processingselection data generated for a particular time slice and/or indicating asame core processing resource to perform the corresponding retrievaland/or recovery of the multiple segments. This can include indicatingthat each entire segment in a set of multiple segments be jointlyretrieved and/or recovered. Alternatively, distinct subsets of datablocks of some or all of the multiple segments can be selected, and thesegment scheduling module 2410 can determine that only particular datablocks of some or all segments in a set of multiple segments be jointlyretrieved and/or recovered.

In some cases, the segment scheduling module 2410 only batches a set ofmultiple physical segments in the same request for retrieval and/orprocessing, for example, by a same particular processing core resource48, when stored in the same memory drive 2440 of segment storage 2442and/or when the corresponding requested data blocks are storedsequentially on the same memory drive 2440. In some cases, the segmentscheduling module 2410 only batches a set of multiple virtual segmentsin the same request for recovery and/or processing, for example, by asame particular processing core resource 48, when the correspondingsegments for retrieval are stored on the same set of nodes 1-K; arestored on an overlapping set of nodes 1-K+r where r is positive and K isthe number of other segments required to recover each segment and/orwhere r is less than or equal to a predefined threshold indicating theamount of overlap is sufficient to constitute batching recovery of theset of segments; and/or are stored on the same set of particular memorydrives or a sufficiently overlapping set of memory drives of the sameset of nodes and/or overlapping set of nodes to facilitate the othernodes own batched retrieval of some or all of the requested segmentsfrom their own segment storage 2442.

FIG. 25E presents a particular example of batching requests to recovermultiple blocks. For a single query, or multiple queries as discussed infurther detail in conjunction with FIGS. 25F-25I, a set of reads tosequential, unavailable data blocks, can be batched, where thesequential data can be recovered in one rebuild action as opposed tobeing rebuilt separately. This can include sequential sub-portions of asame data segment or can include multiple, consecutive data segments.This mechanism of batching can improve database system efficiency byminimizing or otherwise reducing requests sent via the network, and/orby minimizing and/or otherwise reducing processing time that would berequired to recover each of the consecutive data blocks individually.

As illustrated in FIG. 25E, the segment recovery module 2510 can beinstructed by the segment processing module 2430, and/or can beinstructed directly by the segment scheduling module 2410, to recover aset of blocks 1-L of a same virtual segment 3. This can be based on theselection to process segment 3 to service query 2 as illustrated in FIG.25B, in response to a further determination that only blocks 1-L ofsegment 3 are required to service query 2. The set of blocks 1-L canconstitute, for example, a small number of sequential or non-consecutiveblocks of the virtual segment 3. The segment recovery module can send aset of K retrieval requests to retrieve a set of segments 1-K from a setof other nodes 37-1-37-K as illustrated in FIG. 25A. However, in thisexample, only a portion of each segment that are required to, whenprocessed in conjunction with the corresponding portions of allrequested segments 1-K, recover only blocks 1-L of the segment 3 areindicated in the request. As illustrated, only L blocks 1.1-1.L arerequested and retrieved for segment 1; only L blocks 2.1-2.L arerequested and retrieved for segment 2; and only L blocks K.1-K.L arerequested and retrieved for segment K. For example, as discussed inconjunction with FIGS. 25C and 25D, the set of K blocks 1.1-K.1 from theset of K different segments can be utilized to recover data block 1 ofsegment 3; the set of K blocks 1.2-K.2 from the set of K differentsegments can be utilized to recover data block 2 of segment 3; and so onwhere the set of K blocks 1.L-K.L from the set of K different segmentscan be utilized to recover data block L of segment 3.

The set of L blocks can be recovered via a set of L distinct recoveryoperations performed on each of these L block groupings in accordancewith the redundancy storage encoding scheme. Alternatively, inparticular if each set of L blocks are consecutive in each segment, itcan be possible for some redundancy error coding schemes to recover theset of L blocks performed on jointly on the combined and/or concatenatedset of blocks in each segment, where the set of K partial segments ofmultiple concatenated blocks (1.1-1.L)-(K.1-K.L) are processed as jointsub-portions of their respective segments 1.K in accordance with theredundancy error coding scheme to recover blocks 1.L of the segments.These recovered blocks 1.L can be jointly processed to yield one or morecorresponding partial results.

In some cases, other blocks of segment 3 are required, but are recoveredand/or processed separately, for example, because they are a differentset of consecutive blocks that are not consecutive with blocks 1-Land/or because the entirety of required blocks of segment 3 require toomany processing resources to retrieve the corresponding other segmentsand/or process the corresponding other segments jointly. While notdepicted in FIG. 25E, a similar mechanism can similarly be utilized torecover multiple segments stored in the same set of other nodes37-1-37-K.

Batched requests for multiple data blocks and/or multiple segments,whether physical or virtual, can also be intelligently determined by thesegment scheduling module 2410 to service multiple ongoing queries. Inparticular, multiple queries can be serviced simultaneously viautilization of the same physical segment or same virtual segmentdetermined to be required for the multiple queries. As thereconstruction of a virtual segment is a logical construct that is notstored physically on disk, and is only available for the life of thequeries being serviced, the same reconstruction can only servicemultiple queries in this fashion if this reconstruction is forwardedand/or processed in accordance with the multiple queries simultaneouslyand/or within the time frame the recovered segment is available. Thiscoalescing can further improve database system efficiency by minimizingor otherwise reducing recovery processing time that would be required toservice each of these queries via the virtual segment individually, forexample, by reducing and/or minimizing requests sent via the network, asother segments required to reconstruct a particular virtual segment toservice multiple queries need only be retrieved one time for themultiple queries rather than multiple corresponding times; and/or byreducing the number of rebuilding actions, as the segment is onlyreconstructed one time for the multiple queries rather than multiplecorresponding times.

For example, again consider the example presented in FIG. 25B forprocessing of virtual segment 3. As illustrated in FIG. 25F, virtualsegment 3 can be scheduled by the segment scheduler module 2410 to berecovered to process both queries 1 and 2, as both queries 1 and 2require processing of segment 3. This can include assigning sameprocessing resources, such as a same core processing resource 48 and/ora same set of parallel threads of the segment processing module 2430, toretrieve the segment and service both queries utilizing the same segmentin accordance with the operator data 2416. If the node serves to readrows or other raw data of segments for further processing by othernodes, a same partial resultant, such as two identical partialresultants or a single partial resultant corresponding to this read canbe generated by the segment processing module 2430 to be forwarded tothe same node assigned to process these partial resultants as input inaccordance with both queries 1 and 2 and/or to be forwarded to twodifferent nodes, where one node is designated to process this partialresultant of query 1 in accordance with further processing of query 1,and where the other node is designated to process this partial resultantof query 2 in accordance with further processing of query 2.

In some cases, when a particular segment of a segment set of a query,whether virtual or physical, is selected by the segment processingassignment module 2420 for processing, the segment scheduling module2410 determines whether this segment is identified in segment sets ofone or more other queries in the query set. If the segment is alsoindicated in a set of one or more other queries, the segment processingassignment module 2420 can determine whether to coalesce the retrievaland/or processing of the segment for the given query with processing ofthis segment for all of these other queries that require the segment, orfor a selected proper subset of other queries that require the segment.For example, the segment processing assignment module 2420 can determinenot to coalesce retrieval and/or other processing of at least one otherquery that includes the segment based on determining a maximum number ofqueries that can be serviced via processing of the segment at the timeslice based on the processing constraints of the segment processingmodule 2430. In such cases, the segment processing assignment module2420 can select a proper subset of queries that include a same segmentto be serviced via retrieval and/or processing of the segment, forexample, in generating the query processing selection data for a giventime slice, by selecting a predetermined maximum number of queries fromthis set of possible queries that include the same segment to beserviced; and/or by selecting a dynamically determined maximum number ofqueries from this set of possible queries to be serviced, where thedynamically determined maximum is determined based on currentavailability of processing resources and/or based on processing demandsof the corresponding operator data for the set of queries. In somecases, the segment processing assignment module 2420 can determine toselect all queries in the set of possible queries to be serviced in asingle retrieval based on determining the set of possible queriesincludes less than or equal to the predetermined maximum number ofqueries or the dynamically determined maximum number of queries.

In some cases, only a same portion of the data segment and/or portion ofthe segment that includes required sub-portions required by each of themultiple queries is recovered, as discussed in conjunction with FIGS.25C-25E. In such cases, the multiple queries can be identified to haveretrieval and/or recovery of their corresponding segments coalesced whenthey request the exact same portions of the segment, such as a samesubset of data blocks 1-L of the segment. In such cases, this identifiedportion of the segment is retrieved and/or recovered to service each ofthe multiple queries that requires this same portion.

In some cases, even if the portions of the segment required by two ormore different queries differ, an overlapping portion across the two ormore different segments can be utilized to service the two or moredifferent segments as one retrieval and/or recovery. For example,reconsider the example presented in FIG. 25E, where data blocks 1-L ofsegment 3 are recovered to service query 2. In this example, if datablocks 1-L are also required for the use of segment 3 to service query1, they will need to be separately recovered at a different time ifprocessing of segment 3 for queries 1 and 2 was not coalesced into asingle action.

FIGS. 25G-25I illustrates two examples where data blocks 1-L of FIG. 25Eare recovered in a single transaction to service both queries 1 and 2where blocks (1.1-K.1)-(1.L-K.L) only need to be retrieved from othernodes 37-1-37-K one time to service both queries 1 and 2, and/or whereblocks (1.1-K.1)-(1.L-K.L) only need to be processed one time tofacilitate recovery of data blocks 1.L of segment 3. In particular, thesegment scheduler module 2410 can combine the mechanism of batchingsequential reads discussed in conjunction with FIG. 25E with themechanism of coalescing usage of the same data for processing multiplequeries that require usage of the same data.

FIG. 25G illustrates an example of a sequential ordering of data blocksof segment 3 that will be utilized in the example embodiments of FIGS.25H and 25I. As illustrated in FIG. 25G, segment 3 can sequentiallyinclude a plurality of data blocks starting at data block 0, followed bydata block 1. In sequence, data block Q can be included one or more datablocks after data block 1; data block R can be included one or more datablocks after data block Q; data block L can be included one or more datablocks after data block R; and data block S can be included one or moredata blocks after data block R. As segment 3 is utilized as a virtualsegment, the corresponding other segments 1-K in segment 3's segmentgroup used to recover segment 3 can similarly have a same sequentialconfiguration of these data blocks, where data block 1.0-K.0 can beutilized to recover data block 0 of segment 3, and so on, where datablocks 1.S-K.S can be utilized to recover data block S of segment 3.While this illustration indicates an ordering of these data blocks,these data blocks need not be ordered or consecutive with respect tosegment 3 in some embodiments.

FIG. 25H illustrates an example a segment scheduler module 2410 thatselects a set of data blocks, such as a set of sequential data blocks,to service multiple queries based on determining an intersection of datablocks required for multiple queries. In the example of FIG. 25H, query1 requires usage of some or all of data blocks 0-L of segment 3, andquery 2 requires usage of some or all of data blocks 1-S of segment 3.This can be indicated in query set 2415 as illustrated or can otherwisebe determined by the segment scheduler module 2410. The segmentscheduler module 2410 determines that both query 1 and 2 requireprocessing of segment 3 as a virtual segment. As illustrated in FIG.25G, the overlapping, sequential portion of segment 3 that both of thesequeries require includes the set of data blocks 1-L. Alternatively, thedata blocks need not be consecutively stored and/or ordered with respectto the corresponding segment, and batching reads for overlappingportions of required data can still be ideal regardless of the orderingof data. Thus, the set of data blocks 1-L can be any consecutive ornon-consecutive data blocks of segment 3 that are determined to includeexactly the intersection of the first set of consecutive ornon-consecutive data blocks 0-L required by query 1 and the second setof consecutive or non-consecutive data blocks 1-S required by query 2.

The segment scheduler module 2410 can determine the intersection of allrequired data blocks and/or the intersection of sequential required datablocks required to service both queries, and can thus determine thatthis intersection includes data blocks 1-L. In embodiments where onlyintersection portions of data are coalesced, the segment processingassignment module 2420 indicates that data blocks 1-L of segment 3 berecovered to service both queries 1 and 2. The recovery of data blocks1-L can be accomplished as discussed in conjunction with FIG. 25E. Thisparticular recovered portion of segment 3 can be utilized to servicereads or other processing required in operator data of query 1 and query2, respectively, as discussed in conjunction with FIG. 25F to generatethe same or different partial resultant utilizing data blocks 1-L forqueries 1 and 2, for example, via the same core processing resource 48.However, in this example, data block 0 of segment still needs to berecovered and processed to execute query 1, and data blocks (L+1)-Sstill need to be recovered and processed to execute query 2. The segmentscheduler module 2410 can facilitate separate recovery and/or processingof these additional data blocks to service each query separately, forexample, via selection in different time slices and/or via assignment todifferent processing core resources 48 or processing threads. Thismechanism can similarly be applied to three or more queries that allrequire usage of an intersecting set of data, where the intersecting setof data is retrieved and/or recovered to service all of the three ormore queries.

To further reduce the number of network requests and/or processing torecover portions of the segment, the segment scheduler can be operableto identify a union of portions of one or more segments that arerequired to service multiple queries. This can include identifying aunion of all consecutive or non-consecutive portions of a particularsegment that are required by different ongoing and/or pending queries inthe query set 2415. This can further include identifying a union all ofportions across multiple segments stored physically on a same memorydrive that are required by different ongoing and/or pending queries asphysical segments, where reads for some or all of these portions arebatched into one or more batched reads that can service multiplequeries. This can further include identifying a union of all portionsacross multiple segments recoverable via requests to the same set ofother nodes that are required by different ongoing and/or pendingqueries, where external retrieval and/or recovery of some or all ofthese portions of one or more segments required by the same or differentqueries are batched in a single set of external retrieval requests tothis set of nodes and/or where the resulting recovered one or moreportions of one or more segments are utilized to service multiplequeries.

FIG. 25I illustrates an example a segment scheduler module 2410 thatselects a set of data blocks, such as a set of sequential data blocks,to service multiple queries based on determining a union of data blocksrequired for multiple queries. In the example of FIG. 25I, query 1requires usage of some or all of data blocks 1-R of segment 3, and query2 requires usage of some or all of sequential data blocks Q-L of segment3. This can be indicated in query set 2415 as illustrated or canotherwise be determined by the segment scheduler module 2410. Thesegment scheduler module 2410 determines that both query 1 and 2 requireprocessing of segment 3 as a virtual segment. As illustrated in FIG.25G, the union of these distinct, sequential portion of segment 3required to service all portions of segment 3 required for both of thesequeries includes the set of data blocks 1-L. Alternatively, the datablocks need not be consecutively stored and/or ordered with respect tothe corresponding segment, and batching reads for the union of requiredportions of data can still be ideal regardless of the ordering of data.Thus, the set of data blocks 1-L can be again any consecutive ornon-consecutive data blocks of segment 3 that are determined to includeexactly the union of the first set of consecutive or non-consecutivedata blocks 1-R required by query 1 and the second set of consecutive ornon-consecutive data blocks Q-L required by query 2.

The segment scheduler module 2410 can determine the union of allrequired data blocks and/or the union of sequential required data blocksrequired to service both queries, and can thus determine that thisintersection includes data blocks 1-L. In embodiments where the union ofall required portions of data are coalesced, the segment processingassignment module 2420 indicates that data blocks 1-L of segment 3 berecovered to service both queries 1 and 2. The recovery of data blocks1-L can be accomplished as discussed in conjunction with FIG. 25E. Thisparticular recovered portion of segment 3 can be utilized to servicereads or other processing required in operator data of query 1 and query2, respectively, as discussed in conjunction with FIG. 25F to generatethe same or different partial resultant utilizing data blocks 1-L forqueries 1 and 2, for example, via the same core processing resource 48.In this example, processing of both queries may consider or utilize allof data blocks 1-L, even though data blocks R+1-L are not required forquery 1 and even though data blocks 1-(Q minus 1) are not required forquery 2. For example, if these data blocks will automatically befiltered, disregarded, and/or will otherwise not change thecorresponding partial resultants when the operators of operator data2416 are applied, all of the data blocks 1-L can be processed for bothqueries to generate their respective resultants. Alternatively, oncerecovered data blocks 1-L are generated, these recovered data blocks canbe divided into data blocks 1-R and Q-L for processing to generate thepartial resultants for query 1 and query 2, respectively. This mechanismcan similarly be applied to three or more queries that all require usageof data within a determined union of their sets of data, where the unionset of data is retrieved and/or recovered to service all of the three ormore queries. While this example illustrates recovering portions of asingle segment via the single set of external requests 1-K to retrievethe portions of each required segment 1-K, multiple segments included ina union of data required to service multiple queries can be determined,and multiple corresponding required segments for recovery can besimilarly requested in single set of external requests 1-K to retrievethe some or all of each of a plurality of sets of required segments 1-K,where each set in the plurality of sets is utilized to recover one of acorresponding plurality of segments required to service the multiplequeries.

Such embodiments a node 37 that utilizes virtual segments as discussedin conjunction with FIGS. 25A-25I can be implemented in conjunction withany other embodiments of node 37 discussed herein, such as theembodiments previously discussed with regards to FIGS. 24A-24L, forexample, where the segment processing assignment module is operable toselect segments of segment sets for execution, based on driveutilization data 2425, where some of the segments include virtualsegments. In such cases, the drive utilization data 2425 can be utilizedonly determining to select a physical segment to determine whichphysical segment of a subset of segments corresponding to physicalsegments are selected. In other cases, the drive utilization data 2425can be utilized to determine whether to select a physical segment or avirtual segment at a given point in time.

In various embodiments, a node of a computing device has at least oneprocessor and memory that stores executable instructions that, whenexecuted by the at least one processor, cause at least one processingmodule of the node to determine a query for execution and to determine aset of segments that each include a set of rows and/or other datarequired to execute the query. The set of executable instructions, whenexecuted by the at least one processor, can further cause the processingmodule of the node to determine a first segment of the set of segmentsis unavailable and/or to determine the first segment of the set ofsegments be processed as a virtual segment. A locally rebuilt set ofrows or other locally rebuilt data corresponding to the first segmentcan be generated, by the at least one processing module of the node, byutilizing a recovery scheme. For example, the locally rebuild set ofrows can correspond to some or all of the rows included in and/orextracted from a recovered segment, where the recovered segment isgenerated by the at least one processing module of the node inaccordance with a redundancy storage encoding scheme and/or is generatedby utilizing a plurality of other segments retrieved externally fromstorage on a plurality of other nodes. Execution of the query can befacilitated by the at least one processing module of the node byutilizing this locally rebuilt set of rows.

In various embodiments, a node of a computing device has at least oneprocessor and memory that stores executable instructions that, whenexecuted by the at least one processor, cause at least one processingmodule of the node to determine at least one query for execution and todetermine a set of segments that include a set of rows required toexecute the at least one query. The set of executable instructions, whenexecuted by the at least one processor, can further cause the processingmodule of the node to determine at least one virtual segment in the setof segments by identifying segments of the set of segments that areunavailable and/or that are determined to be processed as virtualsegments. At least two consecutive or non-consecutive data portions inthe at least one unavailable segment can be identified, for example,where the at least two data portions are required to execute the atleast one query. A locally rebuilt set of rows or other locally rebuilddata for the at least two consecutive data portions can be generated byutilizing a recovery scheme. For example, the locally rebuild set ofrows can correspond to some or all of the rows included in and/orextracted from a recovered segment and/or recovered portion of asegment, where the recovered segment and/or recovered portion of thesegment is generated by the at least one processing module of the nodein accordance with a redundancy storage encoding scheme and/or isgenerated by processing portions of a plurality of other segmentsretrieved externally from storage on a plurality of other nodes.Execution of the query can be facilitated by the at least one processingmodule of the node by utilizing this locally rebuilt set of rows.

In various embodiments, a node of a computing device has at least oneprocessor and memory that stores executable instructions that, whenexecuted by the at least one processor, cause at least one processingmodule of the node to determine a plurality of queries for concurrentexecution that includes a first query and a second query. A plurality ofsets of segments required to execute the plurality of queries canfurther be determined by the at least one processing module of the node,where the plurality of sets of segments includes a first set of segmentsthat each include a first set of rows required to execute the firstquery and a second set of segments that each include a second set ofrows required to execute the second query. The first set of rows andsecond set of rows can have a null intersection, a non-nullintersection, or can be the same set of rows. The at least oneprocessing module of the node can determine a set of virtual segments inthe plurality of sets of segments by identifying segments of theplurality of sets of segments that are unavailable and/or that aredetermined to be processed as virtual segments. A subset of the set ofvirtual segments can be determined by identifying ones of the set ofvirtual segments that are required to execute multiple ones of pluralityof queries. The subset of the set of virtual segments can include afirst segment included in both the first set of segments and the secondset of segments.

A locally rebuilt set of rows for each of the set of virtual segmentscan be generated by utilizing a recovery scheme. For example, thelocally rebuild set of rows can correspond to some or all of the rowsincluded in and/or extracted from a recovered segment of the set ofunavailable segments and/or recovered portion of a segment of the set ofvirtual segments, where the recovered segment and/or recovered portionof the segment is generated by the at least one processing module of thenode in accordance with a redundancy storage encoding scheme and/or isgenerated by processing portions of a plurality of other segmentsretrieved externally from storage on a plurality of other nodes. Foreach one of the set of virtual segments included in the subset, inresponse to generating the locally rebuilt set of rows, the at least oneprocessing module of the node can facilitate concurrent partialexecution of corresponding multiple ones of the plurality of queries forthe each one of the set of virtual segments included in the subset ofthe set of virtual segments, where a first locally rebuilt set of rowscorresponding to the first segment is utilized to facilitate concurrentpartial execution of the first query and the second query. In somecases, partial execution of the first query and second query can occurin different time slices, but can utilize the same, first locallyrebuilt set of rows, where the first locally rebuilt set of rows is notgenerated for the first and second query via separate performances ofthe recovery scheme.

FIGS. 25J-25L illustrate methods for execution by a node 37. Forexample, the node can utilize at least one processing module of the node37 to execute operational instructions stored in memory accessible bythe node, where the execution of the operational instructions causes thenode 37 to execute the steps of FIGS. 25J, 25K, and/or 25L. The methodsof FIGS. 25J, 25K, and/or 25L can be performed by a node 37 inaccordance with embodiments of node 37 discussed in conjunction withFIGS. 25A-25I, and/or in conjunction with other embodiments of node 37discussed herein.

In the method illustrated in FIG. 25J, step 2550 includes determining aquery for execution. Step 2552 includes determining a set of segmentsthat each include a set of rows required to execute the query. Step 2554includes determining a first segment of the set of segments is to beprocessed as a virtual segment. Step 2556 includes generating a locallyrebuilt set of rows corresponding to the first segment by utilizing arecovery scheme based on determining the first segment is to beprocessed as a virtual segment. Step 2558 includes facilitatingexecution of the query by utilizing the locally rebuilt set of rows.

In various embodiments, determining the first segment is to be processedas virtual segment is based on determining unavailability of a memorydrive that stores the first segment and/or determining a data migrationof the first segment is in progress.

In various embodiments, the first segment is included in a segment groupgenerated by utilizing a redundancy coding scheme. Generating thelocally rebuilt set of rows includes determining a set of segments inthe segment group required to rebuild the first segment, retrieving eachof the set of segments in the segment group from a corresponding one ofa set of memory drives, and utilizing the retrieved set of segments inthe segment group to rebuild the one of the set of segments.

In various embodiments, the method includes foregoing storage of thelocally rebuilt set of rows to disk of the node and/or to segmentstorage of the node, where the locally rebuilt set of rows is notlocally stored once execution of the query is complete. In variousembodiments, the method includes temporarily storing the locally rebuiltset of rows in local cache memory to service the query, and furtherincludes facilitating deletion of the locally rebuilt set of rows fromthe local cache memory in response to completing execution of the query.

In various embodiments, the first segment undergoes a rebuilding processfor a temporal period, where the rebuilding process includesreconstruction of the first segment for storage as a physical segment toa memory drive. The first segment is unavailable during an entirety ofthe temporal period, and the temporal elapses once the first segment isreconstructed in storage as the physical segment in the memory drive.This temporal period begins before the query is determined forexecution, where the temporal period ends after the execution of thequery is facilitated, and where determining the first segment is to beprocessed as a virtual segment is based on determining thatreconstruction of the first segment is incomplete.

In various embodiments, the method includes determining a second queryfor execution after execution of the query is facilitated and during thetemporal period. A second set of segments that each include a second setof rows required to execute the second query is determined, where thefirst segment is included in the second set of segments. The methodincludes determining the first segment is determined to be processed asa virtual segment for the second query, for example, based ondetermining that reconstruction of the one of the set of segments isincomplete. The method includes generating a second locally rebuilt setof rows corresponding to first segment by utilizing the recovery schemeand facilitating execution of the second query by utilizing the secondlocally rebuilt set of rows during the temporal period. In some cases,this second locally rebuilt set of rows is identical to the rebuilt setof rows generated to service the query, but is separately rebuiltbecause the set of rows for the query does not persist in cache memoryand/or for access to service the second query because its utilization infacilitating execution of the query is complete. In other embodiments,the locally rebuilt set of rows is a first subset of a plurality of rowsincluded in first segment, and the second locally rebuilt set of rows isa second subset of the plurality of rows included in first segment, anda set difference between the locally rebuilt set of rows and the secondlocally rebuilt set of rows is non-null.

In various embodiments, the first segment includes a plurality ofportions, such as a plurality of data blocks, that each include adistinct subset of a plurality of rows corresponding included in thefirst segment. Each of the plurality of portions can be independentlyrecoverable via the recovery scheme. The method can further includeidentifying a proper subset of the plurality of portions of the firstsegment that include the set of rows required to execute the query.Generating the locally rebuilt set of rows includes recovering only theproper subset of the plurality of portions of the first segment, whereportions that do not include any rows required to execute the query arenot recovered. In various embodiments, identifying the proper subset ofthe plurality of portions that include the set of rows includesaccessing metadata for the first segment indicating a mapping of theplurality of rows to the plurality of portions.

In various embodiments, a non-transitory computer readable storagemedium includes at least one memory section that stores operationalinstructions that, when executed by a processing module that includes aprocessor and a memory, cause the processing module to determine a queryfor execution and to determine a set of segments that each include a setof rows and/or other data required to execute the query. The set ofoperational instructions, when executed by the at least one processor,can further cause the processing module to determine a first segment ofthe set of segments is unavailable and/or to determine the first segmentof the set of segments be processed as a virtual segment. A locallyrebuilt set of rows or other locally rebuilt data corresponding to thefirst segment can be generated, via the processing module, by utilizinga recovery scheme. For example, the locally rebuild set of rows cancorrespond to some or all of the rows included in and/or extracted froma recovered segment, where the recovered segment is generated by the atleast one processing module of the node in accordance with a redundancystorage encoding scheme and/or is generated by utilizing a plurality ofother segments retrieved externally from storage on a plurality of othernodes. Execution of the query can be facilitated by the at least oneprocessing module by utilizing this locally rebuilt set of rows.

In the method illustrated in FIG. 25K, step 2560 includes determining aplurality of queries for concurrent execution that includes a firstquery and a second query. Step 2562 includes determining a plurality ofsets of segments required to execute the plurality of queries. Theplurality of sets of segments includes a first set of segments that eachinclude a first set of rows required to execute the first query and asecond set of segments that each include a second set of rows requiredto execute the second query. Step 2564 includes determining a set ofvirtual segments in the plurality of sets of segments. Step 2566includes determining a subset of the set of virtual segments byidentifying ones of the set of virtual segments that are required toexecute multiple ones of plurality of queries. The subset of the set ofvirtual segments includes a first segment included in both the first setof segments and the second set of segments. Step 2568 includesgenerating a locally rebuilt set of rows for each of the set of virtualsegments by utilizing a recovery scheme. For each one of the set ofvirtual segments included in the subset, in response to generating thelocally rebuilt set of rows, the method performs step 2570, whichincludes facilitating concurrent partial execution of correspondingmultiple ones of the plurality of queries for the each one of the set ofvirtual segments included in the subset of the set of virtual segments,where a first locally rebuilt set of rows corresponding to the firstsegment is utilized to facilitate concurrent partial execution of thefirst query and the second query.

In various embodiments, the method includes determining a set ofavailable segments in the plurality of segments by identifying segmentsof the plurality of sets of segments that are stored as physicalsegments on a corresponding one of a set of memory drives. The methodfurther includes retrieving the set of available segments from the setof memory drives and facilitating execution of the plurality of queriesby utilizing the retrieved set of available segments and by utilizingthe plurality of locally rebuilt sets of rows. In various embodiments,the method further includes determining an ordering for retrieval of thesegments in the plurality of sets of segments based on the subset of theset of virtual segments.

In various embodiments the first segment includes a plurality ofportions that each include a distinct subset of a plurality of rowsincluded in the first segment. Each of the plurality of portions isindependently recoverable via a recovery scheme. The method can includeidentifying a first proper subset of the plurality of portions thatinclude a subset of rows required to execute the first query, and caninclude identifying a second proper subset of the plurality of portionsthat include a subset of rows required to execute the second query. Themethod can further include determining a union of the first propersubset of the plurality of portions and the second proper subset of theplurality of portions, where the union is a third proper subset of theplurality of portions, and where generating the locally rebuilt set ofrows for the first segment includes rebuilding only the union.Alternatively, the method can further include determining anintersection of the first proper subset of the plurality of portions andthe second proper subset of the plurality of portions, where theintersection is a third proper subset of the plurality of portions, andwhere generating the locally rebuilt set of rows for the first segmentincludes rebuilding the intersection to facilitate concurrent executionof the first query and second query. The method can further includerebuilding, as separate rebuilding processes, for example in other timeslices, the set difference between the first proper subset and the thirdproper subset to facilitate further execution of the first query, andthe set difference between the second proper subset and the third propersubset to facilitate further execution of the second query.

In various embodiments, the method includes determining an intersectionof the first proper subset of the plurality of portions and the secondproper subset of the plurality of portions. The methods can furtherinclude identifying the first segment for inclusion in the subset basedon determining the intersection is non-null, where the first subsetincludes only segments with non-null intersections of proper subsets oftheir plurality of portions required to execute the correspondingmultiple ones of plurality of queries.

In various embodiments, the method includes identifying a second segmentincluded in the first set of segments, the second set of segments, andthe set of virtual segments. The second segment includes a secondplurality of portions that each include a distinct subset of a secondplurality of rows included in the second segment, and each of the secondplurality of portions is independently recoverable via the recoveryscheme. A third proper subset of the second plurality of portions thatinclude a set of rows required to execute the first query is identified,and a fourth proper subset of the second plurality of portions thatinclude a set of rows required to execute the second query is alsoidentified. The method includes determining an intersection of the firstproper subset of the plurality of portions and the second proper subsetof the plurality of portions is null. The second segment is not includedin the subset based on determining the intersection of the first propersubset of the plurality of portions and the second proper subset of theplurality of portions is null. The method further includes, in a firsttemporal period, rebuilding the third proper subset of the secondplurality of portions, and facilitating partial execution of the firstquery based on the third proper subset. The method further includes, ina second temporal period that is distinct from the first temporalperiod, rebuilding the fourth proper subset of the of the secondplurality of portions, and facilitating partial execution of the secondquery based on the fourth proper subset, where the second temporalperiod is distinct from the first temporal period based on determiningthe second segment is not included in the subset.

In various embodiments, the first segment is identified for inclusion inthe subset further based on determining the size of the intersectioncompares favorably to a predetermined intersection size threshold. Thefirst subset includes only segments with intersections of proper subsetsof their plurality of portions required to execute the correspondingmultiple ones of plurality of queries that compare favorably to thepredetermined intersection size threshold. In various embodiments eachof the plurality of segments has a fixed segment size, and thepredetermined intersection size threshold indicates a required minimumproportion of the fixed segment size. In various embodiments, thepredetermined intersection size threshold indicates a required minimumproportion of a union size, where the union size corresponds to adetermined size of a union of the proper subsets of each segment'splurality of portions required to execute the corresponding multipleones of plurality of queries.

In various embodiments, the plurality of queries for concurrentexecution includes a first subset of queries whose execution has beeninitiated based on retrieving at least one segment for each of the firstsubset of queries, and further includes a second subset of queries whoseexecution has not yet been initiated. The first subset of queries andthe second subset of queries are mutually exclusive and collectivelyexhaustive with respect to the plurality of queries for concurrentexecution. The first query is included in the first subset of queries,and the second query is included in the second subset of queries at atime prior to locally rebuilding the first locally rebuilt set of rowscorresponding to the first segment. The method can further includeinitiating execution of the second query in response to locallyrebuilding the first locally rebuilt set of rows corresponding to thefirst segment. The concurrent partial execution of the first query andthe second query utilizing the first locally rebuilt set of rows is afirst one of a plurality of partial executions utilized to facilitateexecution of the second query. The concurrent partial execution of thefirst query and the second query utilizing the first locally rebuilt setof rows occurs after at least one previous partial execution of aplurality of partial executions utilized to facilitate execution of thefirst query.

In various embodiments, the set of virtual segments correspond to a setof unavailable segments, and the set of virtual segments is determinedby identifying ones of the plurality of sets of segments that areunavailable for access as physical segments. In various embodiments, thefirst segment is included in a segment group generated by utilizing aredundancy coding scheme, and generating the locally rebuilt set of rowsincludes determining a set of segments in the segment group required torebuild the first segment, retrieving at least a portion of each of theset of segments in the segment group from a corresponding one of a setof memory drives, and utilizing the retrieved at least a portion of eachof the set of segments in the segment group to rebuild at least aportion of the one of the set of segments.

In various embodiments, the method includes determining a third queryfor execution after the first locally rebuilt set of rows is utilized tofacilitate the concurrent partial execution of the first query and thesecond query. The method includes determining the first segment isrequired to execute the third query, and generating a second locallyrebuilt set of rows the first segment by utilizing a recovery scheme.The second locally rebuilt set of rows includes all of rows of the firstlocally rebuilt set of rows. Second locally rebuilt set of rows isgenerated after the first locally rebuilt set of rows, and is based onthe first locally rebuilt set of rows being unavailable afterutilization in facilitating the concurrent partial execution of thefirst query and the second query. The method includes facilitatingexecution of the third query utilizing the second locally rebuilt set ofrows. In various embodiments, a first temporal period corresponds to thenode's execution of the first query, a second temporal periodcorresponds to the node's execution of the third query, and anintersection between the first temporal period and the second temporalperiod is non-null. The intersection includes at least one partialexecution of the first query by the node, and the intersection includesat least one partial execution of the third query by the node. Forexample, execution of first query by the node is not completed wheninitiation execution of the third query by the node occurs.

In various embodiments, a non-transitory computer readable storagemedium includes at least one memory section that stores operationalinstructions that, when executed by a processing module that includes aprocessor and a memory, cause the processing module to determine aplurality of queries for concurrent execution that includes a firstquery and a second query. A plurality of sets of segments required toexecute the plurality of queries can further be determined by the atleast one processing module, where the plurality of sets of segmentsincludes a first set of segments that each include a first set of rowsrequired to execute the first query and a second set of segments thateach include a second set of rows required to execute the second query.The first set of rows and second set of rows can have a nullintersection, a non-null intersection, or can be the same set of rows.The at least one processing module can determine a set of virtualsegments in the plurality of sets of segments by identifying segments ofthe plurality of sets of segments that are unavailable and/or that aredetermined to be processed as virtual segments. A subset of the set ofvirtual segments can be determined by identifying ones of the set ofvirtual segments that are required to execute multiple ones of pluralityof queries. The subset of the set of virtual segments can include afirst segment included in both the first set of segments and the secondset of segments.

A locally rebuilt set of rows for each of the set of virtual segmentscan be generated by utilizing a recovery scheme. For each one of the setof virtual segments included in the subset, in response to generatingthe locally rebuilt set of rows, the at least one processing module canfacilitate concurrent partial execution of corresponding multiple onesof the plurality of queries for the each one of the set of virtualsegments included in the subset of the set of virtual segments, where afirst locally rebuilt set of rows corresponding to the first segment isutilized to facilitate concurrent partial execution of the first queryand the second query. In some cases, partial execution of the firstquery and second query can occur in different time slices, but canutilize the same, first locally rebuilt set of rows, where the firstlocally rebuilt set of rows is not generated for the first and secondquery via separate performances of the recovery scheme.

In the method illustrated in FIG. 25L, step 2572 includes determining atleast one query for execution. Step 2574 includes determining a set ofsegments that include a set of rows required to execute the at least onequery. Step 2576 includes determining at least one virtual segment inthe set of segments. Step 2578 includes identifying at least two dataportions in the at least one virtual segment. Step 2580 includesgenerating a locally rebuilt set of rows for the at least two dataportions by utilizing a recovery scheme. Step 2582 includes facilitatingexecution of the at least one query by utilizing the locally rebuilt setof rows.

In various embodiments, the at least one virtual segment corresponds toat least one unavailable segment, where the at least one virtual segmentis determined by identifying ones of the set of segments that areunavailable for access as physical segments. In various embodiments, theat least two portions are independently recoverable via the recoveryscheme, and the method further includes determining to batch recovery ofthe at least two portions via a single performance of the recoveryscheme, where the locally rebuilt set of rows for the at least two dataportions by utilizing a recovery scheme are recovered via the singleperformance of the recovery scheme in response to determining to batchrecovery of the at least two portions.

In various embodiments, the at least one segment is included in at leastone segment group generated by utilizing a redundancy coding scheme.Generating the locally rebuilt set of rows can include determining a setof segments in the at least one segment group required to rebuild the atleast two data portions, retrieving at least a portion of each of theset of segments in the at least one segment group from a correspondingone of a set of memory drives, and utilizing the retrieved at least aportion of the each of set of segments in the segment group to rebuildthe at least two data portions based on the redundancy coding scheme. Invarious embodiments, the at least a portion of each of the set ofsegments includes multiple portions, retrieving the multiple portions ofeach of the set of segments includes, for each of the set of segments,sending only one request to retrieve the corresponding multipleportions.

In various embodiments, a first segment of the at least one virtualsegment includes a plurality of sequential data portions. The at leasttwo data portions is a proper subset of consecutive ones of theplurality of sequential data portions, and each of the plurality ofsequential portions is independently recoverable via the recoveryscheme. In various embodiments, the first segment includes at least oneother data portion, where the at least one other data portion and the atleast two data portions are non-consecutive in the plurality ofsequential portions. The method can further include generating a secondlocally rebuilt set of rows for the at least one other data portion byutilizing the recovery scheme, where the locally rebuilt set of rows isgenerated separately from the second locally rebuilt set of rows inresponse to the at least one other data portion and the at least twodata portions being non-consecutive in the plurality of sequentialportions.

In various embodiments, the first segment is included in a segment groupgenerated by utilizing a redundancy coding scheme. Generating thelocally rebuilt set of rows can include determining a set of segments inthe segment group required to rebuild the first segment, where each ofthe set of segments includes a plurality of corresponding data portionsthat can be consecutive or non-consecutive. Generating the locallyrebuilt set of rows can include further include identifying, for each ofthe set of segments, at least two data portions of the correspondingdata portions of the each of the set of segments required to rebuild theat least two data portions of the first segment. Generating the locallyrebuilt set of rows can further include retrieving the subset of thecorresponding data portions of the each of the set of segments from acorresponding one of a set of memory drives, for example, where the setof memory drives are associated with and/or integrated within acorresponding set of other nodes and/or other computing devices.Generating the locally rebuilt set of rows can further include utilizingthe retrieved subset of the corresponding data portions of the each ofthe set of segments to rebuild the at least two data portions based onthe redundancy coding scheme.

In various embodiments, the method includes retrieving the subset of thecorresponding data portions of the each of the set of segments includesgenerating a single read request to each corresponding one of the set ofmemory drives, other nodes, and/or other computing devices. Generatingeach single read request can include by batching a plurality of readrequests to the corresponding one of the set of memory drives, whereeach of the plurality of read requests corresponds to one of the atleast two data portions of the corresponding each of the set ofsegments.

In various embodiments, the at least one segment is included in at leastone segment group generated by utilizing a redundancy coding scheme, andgenerating the segment group includes generating a plurality of dataportion groupings by utilizing the redundancy coding scheme, where eachdata portion grouping includes one data portion, such as a data blocks,from each of a plurality of segments in the segment group, where eachdata portion of a data portion grouping has a same placement in theircorresponding plurality of data portions of their corresponding datasegment, for example, where their corresponding plurality of dataportions is sequential or otherwise has a determined ordering. Each dataportion can be independently recoverable by utilizing the other dataportions in the same data portion grouping via the recovery scheme.

In various embodiments, a first segment of the at least one segmentincludes a plurality of sequential or non-sequential data portions,where the at least two data portions is a proper subset of the pluralityof data portions. The first segment is included in a segment groupgenerated by utilizing a redundancy coding scheme. Generating thelocally rebuilt set of rows includes determining a set of segments inthe segment group required to rebuild the first data segment, where eachof the set of segments includes a plurality of corresponding dataportions. Generating the locally rebuilt set of rows includesidentifying subsets of the corresponding data portions of the each ofthe set of segments required to rebuild the at least two data portions.Generating the locally rebuilt set of rows includes retrieving thesubset of the corresponding data portions of the each of the set ofsegments from a corresponding one of a set of memory drives. Generatingthe locally rebuilt set of rows includes utilizing the retrieved subsetof the corresponding data portions of the each of the set of segments torebuild the at least two data portions based on the redundancy codingscheme.

In various embodiments, identifying the at least two data portions inthe at least one virtual segment includes identifying all of theconsecutive data portions sequential to the least two data portions thatinclude ones of the set of rows required to execute the at least onequery, where a first data portion is stored in sequence immediatelyprior to a first one of the least two data portions, where a second dataportion is stored in sequence immediately after a last one of the leasttwo data portions, and where the first data portion and the second dataportion are not included in the at least two data portions based ondetermining the first data portion and the second data portion do notinclude ones of the set of rows required to execute the at least onequery.

In various embodiments, the at least one query includes a first queryand a second query. A first set of rows is required to execute the firstquery, and a second set of rows is required to execute the second query.The at least two data portions includes a first subset of the first setof rows, and the at least two data portions further includes a secondsubset of the second set of rows. The first subset can be a propersubset of the first set of rows or can include all of the first set ofrows. The second subset can be a proper subset of the second set of rowsor can include all of the first set of rows.

In various embodiments, the first subset of the first set of rows andthe second subset of the second set of rows are mutually exclusive,where a first subset of the at least two data portions includes thefirst subset of the first set of rows, where a second subset of the atleast two data portions includes the second subset of the second set ofrows, and where the first subset of the at least two data portions andthe second subset of the at least two data portions are mutuallyexclusive. In some cases, the first subset and second subset are alsocollectively exhaustive with respect to the at least two data portions.In various embodiments the first subset of the first set of rows and thesecond subset of the second set of rows have a non-zero intersection,and at least one data portion of the at least two data portions includesrows of the first set of rows and the second set of rows. The at leasttwo data portions can include only rows in the non-zero intersection.The at least two data portions can include a union of the first set ofrows and the second set of rows.

In various embodiments, a non-transitory computer readable storagemedium includes at least one memory section that stores operationalinstructions that, when executed by a processing module that includes aprocessor and a memory, cause the processing module to determine atleast one query for execution and to determine a set of segments thatinclude a set of rows required to execute the at least one query. Theset of executable instructions, when executed by the at least oneprocessor, can further cause the processing module to determine at leastone virtual segment in the set of segments by identifying segments ofthe set of segments that are unavailable and/or that are determined tobe processed as virtual segments. At least two consecutive ornon-consecutive data portions in the at least one unavailable segmentcan be identified, for example, where the at least two data portions arerequired to execute the at least one query. A locally rebuilt set ofrows or other locally rebuild data for the at least two consecutive dataportions can be generated by utilizing a recovery scheme. Execution ofthe query can be facilitated by the at least one processing module byutilizing this locally rebuilt set of rows.

FIGS. 26A-26C illustrate embodiments of a node that utilize a processingresource allocation module 2620 to allocate processing resources thatare utilized in processing of segments via the segment processing module2430. The embodiments illustrated in 26A-26C can be utilized toimplement some or all of the plurality of nodes 37 of some or allcomputing devices 18-1-18-n, for example, of the of the parallelizeddata store, retrieve, and/or process sub-system 12, and/or of theparallelized query and results sub-system 13. The embodiments of node 37discussed in conjunction with FIGS. 24A-24K can be utilized to implementany other nodes 37 of database system 10 discussed herein.

As discussed previously, a node can be implemented to read and/orotherwise process a combination of physical segments and virtualsegments to service a query. The processing of virtual segments via therecovery scheme requires multiple external retrieval requests toretrieve a set of other segments from a set of other nodes, and furtherrequires and additional processing to generate the locally rebuildsegment by utilizing the set of other segments. Performing this set ofnetwork reads and subsequent recovery of a requested virtual segmentrequires greater processing resources than retrieving a physical segmentfrom segment storage 2442, for example, via a conventional readoperation. If virtual segments and physical segments are treatedidentically in allocation of processing resources, this higher readlatency of virtual segments will be apparent and can disrupt the readthroughput of a node's execution of a query when these virtual segmentsare being read. Thus, if the amount of processing resources, such as anumber of parallel threads being utilized to service reads, is fixedregardless of whether or not virtual segments are being processed, anoticeable latency can be associated with recovery of a virtual segmentin comparison to retrieval of a physical segment.

The processing resource allocation module 2620 of FIGS. 26A-26C presentsan improvement to the processing of a combination of virtual andphysical segments by masking this latency of virtual segments to achievehigher throughput. This can include generating resource allocation data2628 that indicate processing resources to utilized by segmentprocessing module 2430 to process segments selected in the correspondingsegment processing selection data 2428 based on whether any of thesesegments are to be processed as a virtual segment or a physical segment,for example, as indicated in the query set 2415 and/or as otherwisedetermined by the segment scheduling module 2410. This mechanism can beimplemented by a node's segment scheduling module 2410 to enablesegment-type awareness, where physical segments and virtual segments arescheduled to be handled differently via allocation of different types ofprocessing resources. In particular, the processing resource allocationmodule 2620 is utilized by the node 37 to keep more reads in flight whenvirtual segments are to be processed by increasing the number ofsegments that are being processed in parallel. This includes increasingthe number of parallel threads that are being utilized in the processingof virtual segments. To increase the number of virtual segments beingprocessed, the number of threads dedicated to virtual segments can becorrespondingly increased.

The processing resource allocation module 2620 can be implemented inconjunction with the segment processing assignment module 2420, wherethe segment processing selection data 2428 indicates at least oneportion of at least one segment selected for processing in conjunctionwith at least one corresponding query via the segment processing module2430 for the corresponding time slice as discussed previously, and wherethe segment processing selection data 2428 further indicates resourceallocation data 2628 indicating particular processing resources of thesegment processing module 2430 and/or segment recovery module 2510 thatare allocated for use in retrieval, recovery, and/or processing of thecorresponding at least one portion of at least one segment selected forprocessing in conjunction with at least one corresponding query for thecorresponding time slice. The processing resource allocation module canthus be integrated within the segment processing assignment module 2420and/or can utilize at least one other processing module of the node 37.

The resource allocation data 2628 can indicate a particular processingcore resource 48 or other particular processing module and/or processingresource of the node that is assigned to retrieve, recover, and/orprocess a segment selected in the segment processing selection data 2428for processing a particular query and/or for coalesced processing ofmultiple queries that require the segment as discussed in conjunctionwith FIGS. 25F-25I. For example, the processing resource allocationmodule 2620 can be implemented in the example embodiment presentedpreviously in conjunction with FIGS. 24B-24C, where processing coreresource 48-2 is selected by the processing resource allocation module2620 to retrieve segment 3 for execution of query 2 and/or whereprocessing core resource 48-2 is selected by the processing resourceallocation module 2620 to process segment 5 for query 2 once processingof segment 3 by processing core resource 48-2 is completed.

When a segment is selected for processing in the segment processingselection data 2428, the resource allocation data 2628 can be generatedby the resource allocation module 2620 to indicate whether this segmentis to be processed in parallel with at least one other segment. Aselected set of segments of the segment processing selection data 2428of a particular time slice can be read or otherwise processed by thesegment processing module 2830 within the particular time slice inseries, or in parallel by a corresponding set of parallel threads, basedon a corresponding indication on the resource allocation data 2628. Atany given time, any parallel thread allocated for segment processing canbe utilized to read exactly one segment. Thus, the number of parallelthreads that are allocated by the processing resource allocation module2620 in resource allocation data 2628 for a particular time slice orother time frame can be equal to a number of selected segments to beprocessed in parallel in this time slice or other time frame. Asillustrated in FIG. 26A, the segment processing selection data 2428 canindicate this resource allocation data 2628 indicating the selectednumber of parallel threads, and the segment processing module 2430 canprocess the segments indicated in the segment processing selection data2428 in accordance with the selected number of parallel threadsindicated in the corresponding resource allocation data 2628.

The number of segments to be processed in parallel, dictating thecorresponding number of parallel threads that are allocated forprocessing of segments at a particular time, can be dynamicallyallocated over time as sets of segments are selected for processing bythe segment scheduler module 2410. This dynamic allocation of parallelthreads can be based on whether or not virtual segments are currentlyselected for processing and/or can be an increasing function of a numberof virtual segments that are currently selected for processing. When atleast one virtual segment is selected for processing, the number ofsegments being processed in parallel can be selected to be higher tomask the latency induced by reading of virtual segments.

In some cases, the segment scheduler module 2410 generates segmentprocessing selection data 2428 that indicates a set of segments to beprocessed for a particular time slice that includes a single segment ormultiple segments. A number of parallel threads implemented by segmentprocessing module 2430 are then allocated to facilitate processing ofthe set of segments by the processing resource allocation module 2620.For example, multiple ones of the set of segments can be assigned to asame processing core resource 48 of the segment processing module 2430for concurrent processing via a corresponding set of parallel threads ofthis processing core resource 48. At a given time, each processing coreresource 48 of a node 37 can be processing one or more assigned segmentsvia its own one or more corresponding parallel threads. This number ofparallel threads that will be utilized by a given core processingresource 48 can be selected as a function of the number of virtualsegments in the set of segments scheduled to be processed at this time.In some cases, each virtual segment in the set of segments is allocatedfor processing via its own parallel thread by a core processingresource. The number of parallel threads that are utilized by a singleprocessing core resource 48 at a given time can be equal to and/or anincreasing function of the number of virtual segments assigned to thecorresponding processing core resource 48 at the given time. The numberof parallel threads that are utilized across all processing coreresources 48-1-48-n of a node 37 at a given time can be equal to and/oran increasing function of the number of virtual segments assigned forprocessing across all processing core resources 48-1-48-n at the giventime. For example, at a first time when a first number of virtualsegments are selected for processing, a first number of parallel threadsare allocated for processing the first number of virtual segments. At asecond time when a second number of virtual segments are selected forprocessing, a second number of parallel threads are allocated forprocessing the first number of virtual segments. The second number ofparallel threads can be less than the first number of parallel threadsbased on the second number of virtual segments being less than the firstnumber virtual segments.

In some cases, the processing resource allocation module 2620 can selectand/or allocate a subset of a set of fixed parallel threads of aparticular selected processing core resource and/or can otherwise selectand/or allocate a subset of a set of fixed parallel threads of thesegment processing module 2430 for processing of segments indicated inthe segment processing selection data 2428 to generate the partialresultant of the corresponding query. For example, each processing coreresource 48 and/or the segment processing module 2430 can have a fixedset of parallel threads that can be utilized at any given time forprocessing segments for queries, where a subset of these parallelthreads are allocated for processing of segments for the one or moreparticular queries.

Alternatively, the processing resource allocation module 2620 can selectand/or dynamically generating a set of parallel threads of a particularselected processing core resource 48 and/or otherwise selecting and/orcan dynamically allocate a set of parallel threads of the segmentprocessing module 2430 for processing of the segment to generate thepartial resultant of the corresponding query. For example, the selectedprocessing core resource 48 and/or the segment processing module 2430can have different numbers of the same or different parallel threadsbeing utilized at different time slices for processing segments.Processing core resources 48 and/or the segment processing module 2430can otherwise have different sets of parallel threads being utilized atdifferent time slices for processing segments for queries, where theparallel threads are dynamically allocated by the processing resourceallocation module 2620 and/or where the type and/or amount of processingresources utilized to implement each parallel thread being implementedat a given point in time can be dynamically adjusted by the processingresource allocation module 2620 based on how many parallel threads arerequired at a given point in time and/or based on resource availabilityat a given point in time.

An example of allocating different numbers of parallel threads over timeby one or more different processing core resources 48-1-48-n isillustrated in FIGS. 26B-26C. In FIG. 26B, a plurality of sets ofparallel threads 2652-1-2652-n are implemented by segment processingmodule 2430 at a first time slice t₀ to process a set of segments. Asingle parallel thread is designated as parallel thread 2650, where eachset of parallel threads includes one or more parallel threads 2650. Eachset of parallel threads can be implemented, for example, by acorresponding one of a set of processing core resources 48-1-48-n of thesegment processing module 2430. In some cases, a single set of parallelthreads 2652 is allocated for processing, for example, in cases wheresegment processing module 2430 is implemented via a single processingcore resource 48.

Each set 2652 of the plurality of n sets of parallel threads can beprocessing a corresponding set of segments of one or more queries of thequery set 2415, where the set of parallel threads was previouslyallocated for processing of this set of segments based on resourceallocation data 2628 of segment processing selection data 2428 generatedby the segment processing assignment module 2420 that implements theprocessing resource allocation module 2620 for the time slice or for aprior time slice.

Thus, the plurality of sets of parallel threads are implemented toconcurrently process a corresponding plurality of segments. Thisplurality of segments can include segments from different ones of theset of queries in the query set 2415, where the plurality of segmentsconstitutes a subset of the segments in the segment sets 2418 of thequery set 2415 whose processing has been initiated by segment processingselection data 2428 generated by the segment scheduling module 2410.

In some cases, one or more sets of parallel threads 2652 includesexactly one thread at a particular time slice. For example, exactly onethread can be designated for segment processing by one or moreprocessing core resources to process one or more physical segments inseries, one at a time. In such cases, additional threads are onlyallocated in time slices where virtual segments are selected forprocessing.

In some cases, a set of parallel threads 2652 processes one or more aset of segments in accordance with multiple queries as described inconjunction with FIGS. 25F-25I. In some cases, a set of parallel threadsprocesses multiple data blocks of different segments that were batchedto be processed in tandem in accordance with a single correspondingquery or multiple corresponding queries as discussed in conjunction withFIGS. 25E-25I.

As illustrated in FIG. 26B, a set 2652-1 of the plurality of sets ofparallel threads can include a first number of different parallelthreads, where the first number is equal to a first integer X₁ that isgreater than or equal to one. Another set 2652-2 of the plurality ofsets of parallel threads can include the second number of differentparallel threads, where the second number is equal to a second integerX₂ that is strictly greater than one and is strictly greater than X₁.For example, the second integer X₂ is greater than the first integer toX₁ based on the second processing core resource 48-2 being assigned toprocess a greater number of virtual segments at time slice t₀ than thefirst processing core resource 48-1.

Alternatively or in addition to different processing core resources 48being allocated different numbers of parallel threads for processingsegments at a particular time, the number of parallel threads allocatedto a single processing core resource can dynamically change over timebased on changes in the number of virtual segments assigned forprocessing by the processing core resource at different times. This isillustrated in FIG. 26C. In particular, the set 2652-1 of the pluralityof sets of parallel threads can include a new number of differentparallel threads X₃ for time t₁. In particular, the correspondingprocessing core resource can implement a greater number of parallelthreads for processing its assigned segments indicated in the newsegment processing selection data 2428 for time t₁ based on updatedresource allocation data 2628. This increase of parallel threads from X₁to X₃ can similarly be based on the second processing core resource 48-1being assigned to process a greater number of virtual segments at timeslice t₁ than in time slice t₀. As the number of virtual segmentsdecreases or otherwise changes over time, the number of virtual segmentcan similarly change over time as a function of the number of virtualsegments.

At least one of the parallel threads 2650 of a set 2652 can bedesignated for processing of physical segments. In some cases, exactlyone of the parallel threads 2650 of a set 2652 is designated forprocessing physical segments, for example, where multiple physicalsegments are processed in series by this designated parallel thread. Insuch cases, the number of parallel threads X of a set 2652 can be equalto the number of virtual segments assigned to the correspondingprocessing core resource plus one, where the additional parallel threadis allocated for processing of physical segments by the processing coreresource. In other embodiments, the number of parallel threads 2650allocated to physical segments is a function of the number of physicalsegments assigned for processing, where the number of parallel threads2650 of a set 2652 designated for processing of physical segments can begreater than one. These multiple parallel threads designated forprocessing of physical segments can each process their own distinct setsof allocated physical segments in series. In such cases, total number ofparallel threads 2650 in a set 2652 can be a function of both thephysical segments and the number of virtual segments assigned to theprocessing core resource.

For example, the number of parallel threads 2650 allocated forprocessing physical segments can be determined based on multiplying theassigned number of physical segments by a first factor p₁, and thenumber of parallel threads 2650 allocated for processing virtualsegments can be determined based on multiplying the assigned number ofvirtual segments by a second factor p₂. The factor p₁ can be strictlyless than the second factor p₂, thus inducing higher degrees ofparallelism when virtual segments are added. In some cases, the firstfactor p₁ can be strictly less than one, and the second factor p₂ can beless than or equal to one. In particular, the factor p₂ can be equal toone to dictate that an additional parallel threads is allocated for eachcorresponding virtual segment. In other cases, the factor p₂ can be lessthan one to dictate that one or more parallel threads allocated forvirtual segments may be assigned to process one or more virtual segmentsin series.

The second factor p₂ can be greater than the first factor p₁ by a factorof N, where N is equal to or based on the calculated and/or measuredlatency of processing virtual segments relative to processing physicalsegments. For example, N can be equal to or selected based on an averagevirtual segment read latency divided by an average physical segment readlatency to most effectively mask the latency induced by reads of virtualsegments. The average virtual segment read latency and/or averagephysical segment read latency can be measured and/or calculated based onprior reads of physical and virtual segments, respectively, by segmentprocessing module 2430 historically. In some cases, the factor N isdifferent for different processing core resources based on differentprocessing capabilities. In some cases, the factor N is adjusted overtime based on detecting that the latency ratio between physical andvirtual segments changes over time, for example, due to hardware and/orsoftware updated. As a particular example, the factor N can be based onthe number of external retrieval requests required to retrieve therequired other segments in recovering virtual segments via the segmentrecovery module 2510. For example, the factor N can be equal to, can begreater than, can be a multiple of, and/or can be otherwise adeterministic function of the number K other segments that are retrievedfrom the K other nodes 37-1-37-K, as K distinct external retrievalrequests are required to retrieve the K required segments.

In some cases, a processing core resource 48 is currently processing atleast one segment via at least one corresponding parallel thread. Priorto finishing this processing, a new segment is assigned to be processedby the processing core resource 48 in segment processing selection data2428. If the new segment is a physical segment, this segment can bequeued for processing via an existing one of the parallel threads thatis being utilized to process the at least one segment, for example, oncea parallel thread completes its processing of an existing segment. Inparticular, the resource allocation data 2628 can indicate this segmentbe queued for processing in this fashion in response to this segmentbeing indicated for processing as a physical segment in the query data2415 and/or the segment processing assignment module can wait to assignphysical segments for processing until an existing parallel threadbecomes available to process a new segment by finishing processing of aprevious segment. However, if this new segment is instead a virtualsegment, rather than waiting for the existing parallel threads tocomplete their processing, a new parallel thread can automatically beallocated for processing this new segment via the processing coreresource 48. In particular, the resource allocation data 2628 canindicate that a new parallel thread be added to the existing set ofparallel threads for processing of the segment in response to thissegment being indicated for processing as a virtual segment in the querydata 2415.

In some cases, the segment processing assignment module 2420 can furtherdetermine processor utilization data for each processing core resource48 and/or for the segment processing system 2430 as a whole. Theprocessor utilization data can indicate actual or estimated metrics inone or more recent slices, a current time slice, and/or at least oneupcoming time slice, for each processing core resource 48 and/or for thesegment processing system 2430 as a whole: an amount of availableprocessing resources; a number of current parallel threads beingutilized to process segments; a number of segments currently beingprocessed; a proportion of the current number of segments beingprocessed that are virtual segments; measurements of processing latencyand/or processing performance; estimated processing latency,performance, and/or availability based on previously allocated resourceof previously generated resource allocation data 2628 in one or moreprevious time slices that is determined to and/or estimated to still beallocated due to processing of corresponding segments determined to beand/or expected to be still in progress; or other processinginformation. Some or all of these metrics can be determined forparticular time slices based on sampling processing core resource 48and/or the segment processing module 2430 for metrics of theirprocessing performance and/or availability in some or all time slices.Some or all of these metrics can be determined for particular timeslices based on virtual segments and/or physical segments currentlybeing processed and/or a number of parallel threads currently allocated.

When a virtual segment is selected by the segment processing assignmentmodule 2420 to be processed for a particular time slice, a processingcore resource 48 of the set of processing core resources with a lowestlevel of processing amount of utilization and/or highest level ofavailable utilization indicated by the processor utilization data forthe time slice can be selected by the processing resource allocationmodule 2620 to process the selected virtual segment. A particularprocessing core resource 48 can be otherwise selected for a given timeslice t₀ maximize and/or evenly distribute utilization across allprocessing core resources 48 in a similar fashion discussed with regardsto maximizing and/or evenly distributing utilization across all memorydrive 2440 discussed in conjunction with FIGS. 24D-24K. In some cases,the processing utilization data of a time slice is determined based onand/or is a function of the drive utilization data 2425 of the timeslice and/or of one or more prior time slices.

A processing core resource determined to be processing a greater numberof segments can be determined to have a higher level of processingutilization than a processing core resource determined to be processinga smaller number of segments. A processing core resource determined tobe processing a greater number of virtual segments and/or higherproportion of virtual segments can be determined to have a higher levelof processing utilization than a processing core resource determined tobe processing a smaller number of number of virtual segments and/orsmaller proportion of virtual segments. A processing core resourcedetermined to be utilizing a greater number of parallel threads can bedetermined to have a higher level of processing utilization than aprocessing core resource determined to be processing a smaller number ofparallel threads.

Alternatively in addition, the segment processing assignment module 2420can select virtual segments for processing at particular time sliceswhere the overall utilization of processing resources is lower, and caninstead select physical segments for processing at particular timeslices where the overall utilization of processing resources is higher.For example, virtual segments are only selected for processing in a timeslice if the processor utilization data for the time slice indicates aprocessing utilization level of the segment processing system 2430 as awhole and/or if the processing utilization level of a particularprocessing core resource 48 is below a predetermined and/or dynamicallydetermined processing utilization threshold. If the processingutilization level is not below the processing utilization threshold, novirtual segments are selected for processing during the time slice and aphysical segment is instead selected.

In some cases, the processing utilization threshold is set to a lowerand/or tighter processing utilization threshold when a lower number orproportion virtual segments are included in the query set 2415. This canbe beneficial to only process this low number of virtual segments whenprocessing resources are particularly under-utilized. However, in caseswhere many virtual segments need to be processed, the processingresource allocation module may not have the luxury of assigning virtualsegments for processing in only such under-utilized conditions, as aqueue of remaining virtual segments continually being passed over forselection of physical segments can cause tremendous build up where theseremaining virtual segments would need to be processed all at once. Thus,the processing utilization threshold is set to a higher and/or looserprocessing utilization threshold when a higher number or proportionvirtual segments are included in the query set 2415. This can bebeneficial to ensure that more virtual segments are being processed overtime to accommodate this higher number of virtual segments, even whenprocessing resources are less under-utilized.

As another example, the segment processing assignment module 2420 candetermine a proportion of segments in segments sets 2418 of the queryset 2415 and can evenly distribute the processing of the virtualsegments over the plurality of sequential time slices, for example basedon selecting virtual segments for processing in a proportion of theplurality of sequential time slices that is equal to or substantiallysimilar to the proportion of segments that are virtual segments. As aparticular example, if 10% of the segments in the current query set arevirtual segments, one virtual segment can be selected in every tenthtime slice and/or can be selected for every ten selected segments overany number of time slices. As this proportion of virtual segmentsincreases or decreases over time as new queries are added to the queryset, the proportion of time slices utilized to process the virtualsegments can increase accordingly, where the virtual segments are stillselected to evenly distribute processing of virtual segments over timein accordance with the changing proportion. If the 10% of the segmentsin the current query set increases to 25%, one virtual segment can beselected in every fourth time slice and/or can be selected for everyfour selected segments over any number of time slices Alternatively orin addition, the assignment of virtual segments to core processingresources can similarly be uniformly distributed within a given timeslice and/or over time, where core resources are each assignedprocessing of substantially the same proportion of virtual segments overthe plurality of sequential time slices.

In various embodiments, a node of a computing device has at least oneprocessor and memory that stores executable instructions that, whenexecuted by the at least one processor, cause at least one processingmodule of the node to determine a query for execution and to determine aset of segments required to execute the query. A first proper subset ofthe set of segments is determined by the at least one processing moduleof the node by identifying ones of the set of segments that correspondto physical segments. A second proper subset of the set of segments isdetermined by the at least one processing module of the node byidentifying ones of the set of segments that correspond to virtualsegments. A third proper subset of the set of segments that includes atleast one of the second proper subset of the set of segments areselected by the at least one processing module of the node forprocessing in parallel. The third proper subset of the set of segmentsare processed in parallel to facilitate partial execution of the queryby utilizing a corresponding set of parallel threads of a segmentprocessing module of the node. Each one of the third proper subset ofthe set of segments is processed by utilizing one of the correspondingset of parallel threads.

FIG. 26D illustrates a method for execution by a node 37. For example,the node can utilize at least one processing module of the node 37 toexecute operational instructions stored in memory accessible by thenode, where the execution of the operational instructions causes thenode 37 to execute the steps of FIG. 26C. The method of 26C can beperformed by a node 37 in accordance with embodiments of node 37discussed in conjunction with FIGS. 26A-26C, and/or in conjunction withother embodiments of node 37 discussed herein.

In the method illustrated in FIG. 26C, step 2682 includes determining aquery for execution. Step 2684 includes determining a set of segmentsrequired to execute the query. Step 2686 includes determining a firstproper subset of the set of segments by identifying ones of the set ofsegments that correspond to physical segments. Step 2688 includesdetermining a second proper subset of the set of segments by identifyingones of the set of segments that correspond to virtual segments. Invarious embodiments, the first proper subset and the second propersubset are mutually exclusive and collectively exhaustive with respectto the set of segments. Step 2690 includes selecting a third propersubset of the set of segments that includes at least one of the secondproper subset of the set of segments for processing in parallel. Step2692 includes processing the third proper subset of the set of segmentsto facilitate partial execution of the query by utilizing acorresponding set of parallel threads of a segment processing module ofthe node, where each one of the third proper subset of the set ofsegments is processed by utilizing one of the corresponding set ofparallel threads.

In various embodiments, the set of segments are processed across aplurality of sequential time slices. For each of the plurality ofsequential time slices, the method includes: selecting a subset of theset of segments to be read in the each of the plurality of sequentialtime slices, where the third proper subset of the set of segments areread in a corresponding one of the plurality of sequential time slicesvia the corresponding set of parallel threads; and reading the selectedsubset of the set of segments to facilitate one of a set of partialexecutions of the query utilizing the subset of the set of segments.Different numbers of parallel threads are selected for at least twodifferent ones of the plurality of sequential time slices based on theselected subset of the set of segments in the at least two differentones of the plurality of sequential time slices having different numbersof virtual segments from the second proper subset of the set ofsegments.

In various embodiments, the physical segments are each stored on acorresponding one of a set of memory drives, and processing each segmentof the first proper subset of the set of segments includes retrievingthe each segment from the corresponding one of the set of memory drives.In various embodiments, the virtual segments are not stored on anysingle one of a set of memory drives, and processing each segment of thesecond proper subset of the set of segments includes locally rebuildingthe each segment via a recover scheme. In various embodiments, a set ofprevious physical segments were stored on a corresponding one of a setof memory drives, and the virtual segments of the second proper subsetreplaced the set of previous physical segments based on at least one of:a drive failure or a data migration.

In various embodiments, processing each segment of the second propersubset of the set of segments includes retrieving, for each segment ofthe second proper subset, a corresponding plurality of physical segmentsstored on a set of memory drives of a corresponding one of a set ofother nodes based on sending a set of external retrieval requests to thecorresponding plurality of other nodes. Processing each segment of thesecond proper subset of the set of segments further includes recoveringeach segment of the second proper subset by utilizing a recovery schemebased on the corresponding plurality of physical segments.

In various embodiments, the third proper subset is selected for a firstone of a plurality of sequential time slices and is processed by thecorresponding set of parallel threads and is processed in a first timewindow. The method includes selecting, for a second one of a set ofsequential time slices, a fourth proper subset of the set of segmentsfor processing in series, where the fourth proper subset and the secondproper subset have a null intersection. The method includes processingthe fourth proper subset of the set of segments by utilizing the segmentprocessing module in a second time window, where the first time windowand the second time window have a null overlap. In various embodiments,the fourth proper subset of the set of segments is selected forprocessing in series based on the fourth proper subset including none ofthe virtual segments of the second proper subset of the set of segments.

In various embodiments, the method includes selecting a fourth propersubset of the set of segments that includes another at least one of thesecond proper subset of the set of segments for processing in parallel,where the third proper subset and the fourth proper subset are mutuallyexclusive with respect to the set of segments. The fourth proper subsetof the set of segments is processed in a second time window tofacilitate another partial execution of the query by utilizing anothercorresponding set of parallel threads of the segment processing module.Each one of the fourth proper subset of the set of segments is processedby utilizing one of the same or a different corresponding set ofparallel threads. The first time window and the second time window havea null overlap. In various embodiments, the third proper subset includesa first number of segments, and the fourth proper subset includes asecond number of segments, where the first number of segments is greaterthan the second number of segments. In various embodiments, the firstnumber of segments is selected to be greater than the second number ofsegments based on the third proper subset having a greater number ofvirtual segments than the fourth proper subset. In various embodiments,the first number of segments and the second number of segments are bothgreater than one based on the third proper subset and the fourth propersubset both including at least one virtual segment.

In various embodiments, a fourth proper subset of the set of segments isselected for processing in a second one of the plurality of sequentialtime slices, where the fourth proper subset includes only segments ofthe first proper subset, and where the fourth proper subset areprocessed utilizing a second number of parallel threads that is lessthan the number of parallel threads utilized to process the third propersubset based on including only segments of the first proper subset.where the second plurality of parallel threads is greater than the firstplurality of parallel threads based on the second subset including theat least one segment of the second proper subset. In variousembodiments, the fourth proper subset of the set of segments includes asmaller number of segments than the third proper subset of the set ofsegments based on the fourth proper subset including includes onlysegments of the first proper subset. In various embodiments, a smallernumber of parallel threads are allocated to process the fourth propersubset than the third proper subset based on the fourth proper subset ofthe set of segments including the smaller number of segments than thethird proper subset of the set of segments.

In various embodiments, the method includes determining utilizationdata, such as drive utilization data 2425 or processor utilization data,for each of the plurality of sequential time slices. Each subset of theset of segments for retrieval is selected in each of a plurality ofsequential time slices based on the utilization data determined for theeach of the plurality of sequential time slices. Second utilization datadetermined for the second one of the plurality of sequential time slicesis less favorable than first utilization data determined for the firstone of the plurality of sequential time slices. The third proper subsetof the set of segments is selected to include the at least one segmentof the second proper subset based on the first utilization data beingmore favorable than the second utilization data. In various embodiments,the utilization data is generated based on resource utilization of theset of memory drives and/or resource utilization of the at least oneprocessor of the processing system.

In various embodiments, the method includes determining a plurality ofqueries for execution that includes the query, and further includesdetermining a plurality of sets of segments by determining, for each ofthe plurality of queries, a corresponding set of segments required toexecute the query. The plurality of sets of segments is stored in theset of memory drives. A subset of the plurality of sets of segments isprocessed for each of the plurality of sequential time slices. Onesubset selected for one of the plurality of sequential time slicesincludes segments from different sets of segments of the plurality ofsets of segments.

In various embodiments, a non-transitory computer readable storagemedium includes at least one memory section that stores operationalinstructions that, when executed by a processing module that includes aprocessor and a memory, cause the processing module to determine a queryfor execution and to determine a set of segments required to execute thequery. A first proper subset of the set of segments is determined by theprocessing module by identifying ones of the set of segments thatcorrespond to physical segments. A second proper subset of the set ofsegments is determined by the processing module by identifying ones ofthe set of segments that correspond to virtual segments. A third propersubset of the set of segments that includes at least one of the secondproper subset of the set of segments is selected by the processingmodule for processing in parallel. The processing module processes thethird proper subset of the set of segments to facilitate partialexecution of the query by utilizing a corresponding set of parallelthreads, where each one of the third proper subset of the set ofsegments is processed by utilizing one of the corresponding set ofparallel threads.

FIGS. 27A-27F illustrate embodiments where the segment scheduling module2410 of a node 37 utilizes data ownership information to determine thesegment sets 2418 for the set of queries 2405 in the query set 2415. Theembodiments illustrated in 27A-27F can be utilized to implement some orall of the plurality of nodes 37 of some or all computing devices18-1-18-n, for example, of the of the parallelized data store, retrieve,and/or process sub-system 12, and/or of the parallelized query andresults sub-system 13. The embodiments of node 37 discussed inconjunction with FIGS. 27A-27F can be utilized to implement any othernodes 37 of database system 10 discussed herein.

As discussed previously, multiple nodes 37, such as a particular groupof nodes in a same storage cluster 35, can generate query resultants forthe same query, where the query resultants generated by a storagecluster of nodes 37 in series and/or parallel to ultimately generate thefull resultant of the query. For a given query, a full set of segmentsstored across and/or accessible by the storage cluster of nodes 37executing the query is required. To ensure that the final query resultgenerated via the combined efforts of this storage cluster 35 iscorrect, each one of the set of segments must be processed. Furthermore,each one of the set of segments must be processed exactly once to ensurethat corresponding rows are not duplicated, which could affect the finalresultant of the query. Therefore, for a given query, each segment mustbe retrieved and/or processed by exactly one node in the storage cluster35, such as exactly one node at an IO level of a query execution plan asdiscussed in conjunction with FIGS. 28A-28N.

To ensure that each segment of a query is processed exactly once, allnodes 37 of a storage cluster 35 can store and/or access data ownershipinformation 2710. An example embodiment of the information included indata ownership information 2710 is depicted in FIG. 27A. These nodesresponsible for storing data ownership information 2710 can include allnodes 37 in a group of nodes that are included in an IO level of a queryexecution plan, and/or that are otherwise responsible for performingread steps to read rows in facilitation of query execution. For example,if the storage cluster 35 includes 5 computing devices as illustrated inthe example of FIG. 6 , and if each computing device includes 4 nodesall illustrated in the example of FIG. 7 , the storage cluster caninclude a set of 20 nodes. The data ownership information can include aplurality of node segment sets 2718 for the corresponding plurality ofnodes in the storage cluster. As illustrated in FIG. 27A, a plurality ofnodes 37-1-37-W of the storage cluster can each have a correspondingnode segment set 2718 of a corresponding plurality of node segment sets2718-1-2718-W. Each node segment set can indicate the full set ofsegments that are owned by the segment. As used herein, a node's“ownership” of a segment corresponds to a node being assigned to readand/or process this segment in accordance with processing queries and/orthat the node is otherwise responsible for retrieval, recovery, and/orprocessing of the corresponding segments in its execution of queries inits query set 2405.

Each node segment set 2718 can further indicate whether thecorresponding node is responsible for processing these segments asvirtual or physical segments. Some or all the segments in a node segmentset 2718 for a particular node can be physical segments that aredirectly accessible by the node via its segment storage 2442. Some orall of the segments in a node segment set 2718 for a particular node canbe virtual segments that are accessible via a recovery scheme. Thus, anode's “ownership” of some segments can correspond to virtual segmentsthat are not stored by the node in its own segment storage 2442.

In the example presented in FIG. 27A, node 37-1 owns a plurality ofsegments that include segments 1, 2, 3, 4, 5, 6, X, Y, and Z; node 37-2owns a plurality of segments that include segments 7-15; and node 37-Wowns a plurality of segments that include segments 16-24. These segmentnumbers are included to label the segments, and do not necessarilyindicate any ordering of these segments. In this example, the nodesegment set of node 37-1 indicates segments 3, 4, and Y are owned bynode 37-1 virtual segments, and the node segment set of node 37-2indicates segments 9, 10, and 11 are owned by node 37-2 as virtualsegments.

The nodes 1-W can process their queries by generating correspondingsegment sets 2418 of incoming queries. In particular for a given query2405 to be processed by a node 37, it can determine the correspondingsegment set 2418 to include all required segments for the given querythat are owned by the node as indicated by the data ownershipinformation 2710, and only the required segments for the given querythat are owned by the node the data ownership information 2710. The nodecan further determine whether each particular segment in the segment setis to be processed as a physical or virtual segment based upon itscorresponding indication in the data ownership information 2710.

The data ownership information 2710 can indicate, in exactly one nodesegment set 2718, each one of the full set of segments owned by thecorresponding storage cluster 35, such as the full set of segments thatare stored by the storage cluster and/or the full set of segments thecorresponding storage cluster is responsible for. Thus, the plurality ofnode segment sets of a storage cluster's data ownership information 2710can be mutually exclusive and collectively exhaustive with regards tothe full set of segments owned by the corresponding storage cluster 35.In some cases, not all of the storage cluster's full set of segments arecurrently stored by the storage cluster 35, for example, where they areonly recoverable as virtual segments due to the corresponding physicalsegments being unavailable.

The data ownership information 2710 can correspond to a particularstorage cluster 35 and can include node segment sets for every one ofits node 37-1-37-W, such as a distinct set of 20 nodes. Each storagecluster of a plurality of different storage clusters in the databasesystem 10, such as the plurality of storage clusters 35-1-35-z, can eachhave its own corresponding data ownership information for its owncorresponding set of nodes. Queries can be processed by nodes of asingle storage cluster 35 and/or via nodes of multiple storage clusters35, for example, if they include segments in data ownership information2710 of different storage clusters. Thus, to maintain query correctnessacross multiple storage clusters, the plurality of full sets of segmentsof the corresponding plurality of storage clusters can be mutuallyexclusive and collectively exhaustive with regards to all segments thatare stored and/or recoverable by the database system 10 as a whole.

The portion of data ownership information 2710 accessible by aparticular node can indicate only the proper subset of the full set ofsegments stored nodes in the storage cluster that are owned by theparticular node. For example, each node 37-1-37-W may store, access,and/or be able to determine its own node segment set 2718. In suchcases, the particular node may not have knowledge of which other nodes37 in the storage cluster 35 store particular other segments that aren'towned by the particular node. Alternatively, as the particular node 37may need to access segments stored by particular other nodes as part ofa recovery scheme utilized in processing virtual segments of their nodesegment set 2718 as discussed in conjunction with FIGS. 25A-25L, eachnode 37 in the storage cluster can store, access, and/or otherwisedetermine the some or all of the full data ownership information 2710.

In this example presented in FIG. 27A, node 37-1 can be implemented bythe node 37 illustrated in FIG. 25B. The data ownership information 2710is utilized by the segment scheduling module 2410 of node 37-1 todetermine that segments 1, 2, 3, 4, 5, 6, X, Y, and Z are to beprocessed in queries accordingly, if required by particular queries 2405in the node's query set 2415. For example, this node 37-1 determines itssegment set 2418 for query 2 includes segment 3, segment 5, and segmentY in response to first determining a full set of segments required forexecution of query 2, and by next determining its own segment set 2418as a proper subset of this full set of segments required for executionof query 2, where other segments in this full set of segments requiredfor execution of query 2 are processed by other nodes in the storagecluster 35. In particular, segments 3, 5 and Y are identified in thisproper subset because they are included in the full set of segmentsrequired for execution of query 2, and are further included in node37-1's node segment set 2718. Even if node 37-1 determines that othersegments, such as segment 7, is required for execution of query 2,segment 7 will not be included in node 37-1's segment set 2418 for query1 because it is not owned by the node, and will instead be processed bynode 37-2 in accordance with query 2.

Continuing with this example, node 37-1's segment set indicates segments1, 2, 5, 6, X, and Z are to be processed as physical segments, and thatsegments 3, 4, and Y are to be processed as virtual segments. This canbe due to the data ownership information 2710 being determined inresponse to and/or during the outage of memory drive 2440-2 that storessegments 3, 4, and Y. For example, a previous version of data ownershipinformation determined before the outage of memory drive 2440-2 may haveindicated that segments 3, 4, and Y were owned by node 37-1 as physicalsegments due to their availability in segment storage.

Thus, the data ownership information 2710 can change over time, whereupdated versions of the data ownership information 2710 can be generatedand utilized, for example, over one or more ones of the plurality ofsequential time slices. In particular, data migration within the storagecluster or between different storage clusters, drive outages, or otherchanges in availability of particular segments can cause segments infull set of segments in a storage cluster to change ownership indifferent versions of the data ownership information over time; tochange from being owned by the same or different node as a virtual orphysical segment in different versions of the data ownership informationover time; to include new segments added to the storage cluster, forexample, as new data to the database system 10 and/or as migrated datafrom a different storage cluster, in different versions of the dataownership information over time; to drop the inclusion of segmentsremoved from the storage cluster, for example, based on being migrateddata to a different storage cluster and/or being deleted from thedatabase system entirely, in different versions of the data ownershipinformation over time; and/or to otherwise change over time.

Alternatively, the same storage cluster 35 will always maintainownership of its full set of segments over time to guarantee consistencyacross multiple storage clusters while not requiring any coordinationacross multiple storage clusters, where changes in a storage cluster'sdata ownership information only includes changes in distribution ofownership across nodes within the storage cluster of its fixed full setof segments. In particular, as each single storage cluster stores allsegments within each segment group for segments stored by the storagecluster, ownership of unavailable segments of the storage cluster can bemaintained as virtual segments assigned to nodes in the storage clusterfor recover via retrieval of other segments 1-K from other nodes 1-K inthe same storage cluster.

Each version of the data ownership information 2710 can be tagged orotherwise be associated with a corresponding ownership sequence number(OSN) 2720. As illustrated in FIG. 27A, the data ownership information2710 is tagged with OSN 7, for example, to indicate that it is theseventh version of the data ownership information 2710, where the OSNincrements with each corresponding updated version of the data ownershipinformation 2710 over time. Alternatively, the OSN can be any uniqueidentifier that distinguishes the corresponding version of dataownership information 2710 from other versions.

Rather than necessitating global coordination and/or single entityresponsible for assignment and sharing of data ownership information asnew versions are generated over time, each new version of the dataownership information of a particular storage cluster 35 can begenerated via a consensus protocol, which can be executed by some or allnodes 37 in a storage cluster participating in the consensus protocol,where the shared state mediated via the consensus protocol indicates themost updated ownership information. This mechanism improves databasesystems by guaranteeing consistency of data ownership information acrossnodes for usage in queries while not requiring global coordination.

For example, as illustrated in FIG. 27B, a plurality of consensusprotocol executions 2750 can be performed via the nodes 37-1-37-W in astorage cluster 35 over time to generate a corresponding plurality ofversions of data ownership information 2710. For example, as illustratedby timeline 2740, a first consensus protocol execution 2750-1 can bemediated across nodes in the storage cluster during timespant_(1.0)-t_(1.1) to generate a corresponding first version of dataownership information 2710-1. For example, the first consensus protocolexecution 2750-1 can be initiated at time t_(1.0) by one or more nodesin the storage cluster 35, and the first consensus protocol execution2750-1 can be completed, for example, where some or all nodes in thestorage cluster have determined and/or can access the resulting dataownership information 2710-1, at t_(1.1). At some time after t_(1.1), orperhaps instead at some time before the first the first consensusprotocol execution 2750-1 is complete but after the first consensusprotocol execution 2750-1 is initiated, a second consensus protocolexecution 2750-2 can be mediated across the nodes in the storage cluster35 to generate to generate a corresponding second version of dataownership information 2710-2 during timespan t_(2.0)-t_(2.1). Similarly,a third consensus protocol execution 2750-3 can be mediated across thenodes in the storage cluster 35 to generate to generate a correspondingthird version of data ownership information 2710-3 during timespant_(3.0)-t_(3.1), and this process can continue over time where consensusprotocol executions 2750 are performed to generate corresponding dataownership information 2710 over time. Data ownership information 2710-1,2710-2, and 2710-3 are tagged with their respective OSNs 2720 withvalues of 1, 2, and 3, respectively, or otherwise indicating theordering of the revision with respect to the other revisions.

As discussed herein, consider the times t_(1.0), t_(2,1), t_(3.1), oftimeline 2740 as the times where the resulting corresponding versions ofdata ownership information 2710-1, 2710-2, 2710-3, . . . 2710-i,respectively, are available for utilization by the nodes 37 in thestorage cluster for query execution as a result of consensus protocolexecutions 2750-1, 2750-2, 2750-3, . . . , 2750-i being completed acrossthe set of nodes in the storage cluster 35, where i is any ith iterationof executing the consensus protocol to generate a corresponding ithversion of the data ownership information 2710. The OSN for any ithversion of the data ownership information can be tagged with arespective OSNs 2720 indicating that the version is the ith version inthe ordering, for example, where the value of the OSN 2720 is equal toor otherwise indicates the value of i.

As illustrated in FIG. 27B, the consensus protocol can be executed viaconsensus protocol communications generated by nodes 37 and/or receivedand processed by nodes 37. For example, each node can implement a dataownership consensus module 2730, for example, by utilizing at least oneprocessing module of the node. The data ownership consensus module 2730can be utilized by each corresponding node 37 to generate consensusprotocol communications in accordance with the storage cluster'sexecution of the current consensus protocol for transmission to one ormore other nodes in the storage cluster in accordance with the storagecluster's execution of the current consensus protocol, for example, viasystem communication resources 14. The data ownership consensus module2730 can be utilized by each corresponding node 37 to receive and/orprocess consensus protocol communications, generated by other nodes inthe storage cluster 35 in accordance with the storage cluster'sexecution of the current consensus protocol. The consensus protocol canbe a leader-mediated consensus protocol. Execution of the consensusprotocol can include election or other determination of a leader by oneor more nodes, voting by one or more nodes, and/or ultimately arrivingat a consensus based on the voting by the one or more nodes to generateand/or communicate the resulting data ownership information.

One or more nodes can initiate a revision of the data ownershipinformation 2710 by initiating a new execution of the consensusprotocol, for example, in response to determining a changed data storagecondition such as a drive outage, a full rebuild of data beingcompleted, a migration being initiated or completed, current orscheduled upcoming data unavailability, or another change. Alternativelyor in addition, new executions of the consensus protocol to generaterevised data ownership information 2710 can occur at scheduled and/orpredetermined times.

Because data ownership information is local only to a particular storagecluster 35, each storage cluster of a small number of nodes can executethe consensus protocol amongst themselves, rather than requiringconsensus or other coordination across all nodes in the database system.Each of the storage clusters in the plurality of storage clusters35-1-35-z can independently generate their own iterative revisions oftheir own data ownership information over time in their own timeline2740, where at any given point in time, different storage clusters mayhave independently generated a different number of revisions of theirdata ownership information. This improves database systems by ensuringthat the execution of the consensus protocol remains scalable, whereonly local coordination is required to determine data ownershipinformation, while ensuring that all segments across different storageclusters 35 has consistent ownership information.

As revised data ownership information is determined by particular nodesover time, most recent versions of the data ownership information 2710can be implemented to execute incoming queries. However, if the nodewere to immediately adopt the most recent data ownership information forsegment processing in executing queries in query set 2415, queries couldbe processed improperly. In particular, as an individual node executes aquery over a span of time, if the node changes its segment setdetermined for the query based on a more recent versions of the dataownership information 2710 mid-execution, some segments needed forexecution of the query across all nodes can be missed and/or duplicated.Furthermore, multiple nodes can be executing the same query withinslightly different time spans based on their own segment schedulermodule's initiation of execution of a particular query. Alternatively orin addition, the most recent data ownership information can be receivedand/or determined by the different nodes at slightly different times. Asglobal coordination is not utilized and as nodes independently executequeries via the segments they determine to own, a mechanism to ensureall nodes execute each given query with the same data ownershipinformation is required.

FIGS. 27C-27F illustrate an example of an embodiment of the presentinvention where nodes in a storage cluster utilize OSNs 2720 tagged toand/or determined for each query 2405 in the query set 2415 to determinewhich corresponding one of a plurality of data ownership informationversions 2710 generated via the storage cluster's execution of theconsensus protocol over time will be utilized to determine thecorresponding segment set 2418 for each query 2405.

FIG. 27C illustrates a particular example of timeline 2740 to illustratethe temporal relation between a series of events occurring at particularpoints in time and/or time spans t₀-t₁₀. At a point in time t₂, dataownership information 2710 with OSN 7 is generated. For example, theexecution of the consensus protocol can be completed at time t₂ torender the resulting data ownership information. This particular versionof the data ownership information may have been generated in response toa failure of memory drive 2440-2 of node 37-1 at time t₀. In thisexample, node 37-1 may have initiated the consensus protocol shortlyafter time t₀ in response to detecting the failure and/or before time t₀in response to this outage being scheduled. Alternatively or inaddition, another node 37 in the storage cluster may have detected thefailure of the memory drive, for example, based on failing to retrievedata stored in this memory drive as part of a recovery scheme forrecovering one of their owned virtual segments. Alternatively, thestorage cluster may have otherwise determined to generate data ownershipinformation 2710 with OSN 7 in response to this failure.

This failure of memory drive 2440-2 can correspond to the particularexample discussed in conjunction with FIG. 25B, where data ownershipinformation 2710 with OSN 7 indicates that node 37-1 maintains ownershipof some or all of the segments of memory drive 2440-2, but thedesignation has changed to virtual segments as these segments areunavailable as physical segments. The data ownership information 2710with OSN 7 of this example is illustrated in FIG. 27D. In particular,segments 3, 4, and Y, which were stored on 2440-2 of 37-1, are indicatedas virtual segments, for example, changing from designation as physicalsegments owned by 37-1 in prior data ownership information 2710 with OSN6.

Timeline 2740 of FIG. 27C indicates a span of time in which a full arebuild of the memory drive 2440-2 of node 37-1 takes place to recoverand store some or all segments of memory drive 2440-2 as physicalsegments in one or more memory drives of the segment storage of anothernode 37-2. For example, this is initiated at time t₁, for example, basedon determining of the memory drive 2440-2 failed at time t₀. Theexecution of the consensus protocol for the data ownership informationof OSN 7 may have been initiated before or after this full rebuildbegan. However, as the full rebuild is lengthy and/or because the fullrebuild was not completed when the initiation of data ownership theconsensus protocol for generating the data ownership information of OSN7 occurred, the data ownership information of OSN 7 reflects that thesesegments are not available physically and assigns ownership as virtualsegments.

Timeline 2740 also illustrates that after the full rebuild of memorydrive 2440-2 is completed, a next version of data ownership information2710 is generated, tagged to OSN 8. For example, the execution of theconsensus protocol for this next version can be completed at time t₇ torender the resulting data ownership information. In this example, node37-2 or another node of the storage cluster may have initiated thisconsensus protocol shortly after time t₄ in response to determining thefull rebuild is completed and/or that the corresponding segments areagain available as physical segments.

Data ownership information 2710 of OSN 8 reflects the availability ofthese segments as physical segments of segment storage 2442 of node 37-2by indicating assignment of some or all of these newly rebuilt segmentsto node 37-2 as physical segments. For example, as illustrated in FIG.27D, the data ownership information 2710 with OSN 8 indicates thatsegments 3, 4, and Y have been added to node 37-2's node segment set2718-2 as physical segments. Furthermore, as segments cannot be owned bymultiple nodes, these segments are removed from node 37-1's node segmentset 2718-1. The “X”s indicated in FIG. 27D serve to illustrate the priorinclusion of these segments in node segment set 2718-1 of data ownershipinformation 2710 with OSN 7 have been removed in the next revision,where segments 3, 4, and Y are not included in the node segment set2718-1 of the data ownership information 2710 with OSN 8.

This example serves to illustrate how the tagging of OSNs to particularqueries can ensure that, despite this timeline of changing dataavailability circumstances that could lead to confusion regarding whichsegments are owned by a node at particular times and more specifically,for different queries being executed by the node at the same time. Thisimproves database systems by ensuring that, despite differentconcurrently running queries at a given time by a given node, anddespite the concurrent, independent execution of each concurrentlyrunning query across multiple nodes in the storage cluster, queryaccuracy of every query is guaranteed because all nodes will utilize thesame data ownership information for any given query, even if differentownership information is utilized at a particular time for different,corresponding concurrently running queries. Thus, different queries withdifferent OSNs can be safely running in parallel by each of a set ofmultiple nodes.

A first query, query 1, can be executed by the storage cluster 35 fromtime t₃-t₆. Time t₃ can correspond to a time at which query 1 wasreceived and/or at which at least one node initiated a partial executionof query 1. Time t₆ can correspond to a time at which execution of query1 by all nodes in the storage cluster assigned to execute query 1 hascompleted. While execution spans of different nodes in the storagecluster may be different based on their own implementation of theirsegment scheduling module 2410, for the purposes of this example, assumethat the time frame that both particular nodes 37-1 and 37-2 executedquery 1 started between t₂ and t₄ and ended between t₅ and t₇.

A second and third query can similarly be executed by the storagecluster from times t₅-t₉ and times t₈-t₁₀, respectively. Again, forpurposes of this example, assume that the time frame that bothparticular nodes 37-1 and 37-2 executed queries 2 and 3 started andended substantially close to these times relative to other pointsillustrated in the timeline 2740 of this example. Also note that asillustrated, the execution of queries 1, 2, and 3 is overlapping, toreflect the concurrent execution of multiple queries implemented by thestorage cluster and to further reflect the concurrent execution ofmultiple queries implemented by each node 37 in the storage cluster.

The execution of these queries by node 37-1 in accordance withdetermined OSNs for these queries is reflected in FIG. 27E, and theexecution of these queries by node 37-1 in accordance with determinedOSNs for these queries is reflected in FIG. 27F. FIGS. 27E and 27Fillustrate nodes 37-1 and 37-2, respectively, that each implement asegment set generating module 2765. The segment set generating module2765 can be implemented by any node 37 in the database system 10, forexample, implemented by the segment scheduling module 2410 of the nodeand/or otherwise implemented utilizing at least one processing module ofthe node. The segment set generating module 2765 can be operable togenerate some or all segment sets 2418 for corresponding queries 2405 ofquery set 2415 of the node that is utilized by the segment schedulingmodule 2410 to generate the segment processing selection data 2428dictating the ordering in which segments of different queries will beprocessed by the node, as discussed in conjunction with FIG. 24A. Thesegment set generating module 2765 can be operable to update this queryset 2415 as new queries are received for execution over time, wheresegment sets 2418 for each incoming query 2405 are generated by thesegment set generating module 2765 for inclusion in query set 2415.

In particular the segment set generating module 2765 can determine thesegment set for each incoming query based on the OSN assigned to and/ordetermined for each incoming query. For a given query with acorresponding tagged OSN, segment set generating module 2765 can accessits node segment set 2718 in the data ownership information 2710 withthe corresponding OSN. In particular, each node can access locallystored, retrievable, or otherwise determinable historical data ownershipinformation 2760 that indicates a plurality of versions, such as asubset of all versions over time corresponding to the most recentversions still determined to be relevant and/or all versionshistorically. Alternatively, if incoming queries are assigned an OSN tagfor the most recent data ownership information 2710, only the mostrecent data ownership information 2710 need be stored and/orretrievable, as the necessary information for prior data ownershipinformation 2710 with prior OSNs can be already reflected in previouslygenerated segment sets 2418 for other queries still being executed inaccordance with older data ownership information 2710.

While not illustrated, the historical data ownership information can berepresented as a plurality of (segment, OSN) pairs for the node. Thesegments of the node's node segment set 2718 in the data ownershipinformation for a given OSN can be each be indicated in a correspondingset of (segment, OSN) pairs with the given OSN. In executing a querytagged with a given OSN, only segments included (segment, OSN) pairsthat reflect the corresponding OSN are utilized. Thus, the node segmentset for a given OSN is derived from and/or represented as all of thesegments included in the node's (segment, OSN) pairs with the given OSN.

The particular node segment set 2718 in the data ownership information2710 with the OSN tagged to an incoming query can be utilized togenerate the segment set 2418 for this incoming query. In particular,the segment set 2418 of this incoming query must be a subset of the nodesegment set 2718 of the data ownership information 2710 with an OSN thatmatches that of the incoming query or otherwise compares favorably tothe incoming query. In some cases, the segment set 2418 of this incomingquery is only a proper subset of the corresponding node segment set2718, for example, based on one or more nodes being determined not to benecessary to process the query and/or not being included in the querydomain of the query. Filtering the node segment set 2718 to generate thecorresponding segment set 2418 can include extracting information fromthe query itself to determine which particular proper subset of segmentsare required.

The OSN 2720 assigned to each query can be received by the node 37 inconjunction with receiving a request to execute the query and/or can bereceived in conjunction with the query itself, for example, where theOSN is generated by another entity of the database system 10 and/or ofthe corresponding storage cluster 35 and is sent to and/or accessible byall nodes executing the query in conjunction with information regardingthe query for execution itself. The OSN of a given query can bealternatively determined by each node based on the query, for example,by comparing a timestamp of the query to timestamps associated with eachof the plurality of versions, and selecting the most recent one of theplurality of OSN versions that has a corresponding timestamp indicatingit was generated prior to the query and/or indicating it can be utilizedon incoming queries after a particular point in time, such as t_(i.1).The node can alternatively perform another deterministic function on agiven query to determine the OSN assigned to the given query.

The mechanism utilized by a node to determine a query's OSN can be thesame for all nodes in the storage cluster 35 to ensure that a givenquery executed by multiple nodes in the storage cluster 35 will assign anode the same OSN, thus ensuring a correct query result as each requiredsegments will be read by a corresponding node, and as each requiredsegment will be read by only one node.

Furthermore, if multiple storage clusters are required for execution ofa query, nodes in different clusters will thus assign a given querydifferent OSNs for corresponding different data ownership information oftheir storage cluster. However, despite different storage clusters beingon different revisions of their data ownership data and mediating theirdata ownership data separately, query correctness can still beguaranteed where each required segment is read once and exactly once solong as nodes in the same storage cluster each utilize the same one oftheir revised data ownership information 2710 for the query, and so longas each storage cluster maintains ownership of their own fixed, full setof nodes in their set of revisions over time.

The generation of segment sets 2418 based on an OSN determined for thequery to adhere to a corresponding version of the data ownershipinformation ensures that a particular version of the data ownershipinformation is used by every node in the storage cluster for executionof the query, and persists for the life of the query regardless of newversions of the data ownership information that are determined while thequery is executing and/or regardless of changes in storage circumstanceswhile the query is executing.

In particular, in this example, all nodes in the storage cluster,including nodes 37-1 and 37-2, determine to execute query 1 by utilizingthe data ownership information with OSN 7, to execute query 2 byutilizing the data ownership information with OSN 7, and to executequery 3 utilizing the data ownership information with OSN 8. Thesedetermination of OSNs tagged to each query can be based on determiningthat the most recent OSN when each query was received and/or beganexecuting. Queries 1 and 2 were received and/or began executing withdata ownership information with OSN 7 being the most recent, asillustrated in timeline 2740 of FIG. 27C, and are tagged with OSNaccordingly. The data ownership information was updated to the dataownership information with OSN 8 prior to receiving and/or initiatingexecution of query 3, so query 3 can be tagged to OSN 8.

Despite the full rebuild of segments of memory drive 2440-2 during query1's execution, all nodes will maintain utilization of OSN 7 for theentirety of query 1's execution, and thus virtual segments of thismemory drive will still be utilized by node 37-1 for the entirety ofquery 1's execution, and node 37-2 will not utilize these segments,despite being rebuilt and available to node 37-2, for its own executionof query 1.

Assume in this example that queries 2 and 3 require utilization ofidentical segments, and thus, if executed by the same node with the sameOSN, would have identical segment sets 2418 for that node. However, inthis example, each of these queries are tagged to different OSNs, andthus have different segment sets. As illustrated in FIG. 27E, for query2, node 37-1 utilizes a segment set 2418 with segments 3, 4, and Yincluded as virtual segments, but these segments are not included innode 37-1's segment set 2418 for query 3, based on these nodes beingincluded in node 37-1's node segment set 2718 for OSN 7, but not OSN 8,and based on query 2 being executed under OSN 7 and query 3 beingexecuted under OSN 8.

Meanwhile, as illustrated in FIG. 27F, for query 2, node 37- utilizes asegment set 2418 that does not include segments 3, 4, and Y, but thesesegments are not included in node 37-2's segment set 2418 for query 3,based on these nodes being included in node 37-2's node segment set 2718for OSN 8, but not OSN 7, and based on query 2 being executed under OSN7 and query 3 being executed under OSN 8. In particular, despitesegments 3, 4, and Y being available as physical segments to node 37-2prior to query 2 being executed, these segments are not utilized forexecution of query 2 because it is tagged to OSN 7 as the new dataownership information is not yet generated. Furthermore, despite the newownership information with OSN 8 being generated during query 2'sexecution, both node's 37-1 and 37-2, as well as all other nodes in thestorage cluster, will maintain utilization of OSN 7 for query 2 for theremainder of query 2's execution. Finally, note that in a periodtemporal period that includes the time span from t₈-t₉, nodes 37-1 and37-2 are each concurrently executing multiple queries by utilizingdifferent OSNs for these multiple queries during this temporal, wherequery 2 is being executed during the time span from t₈-t₉ utilizingprior data ownership information with OSN 7, and where query 3 isconcurrently being executed during the time span from t₈-t₉ utilizingupdated data ownership information with OSN 8.

FIGS. 27G-27J illustrate an extension of the example of FIGS. 27C-27F.As illustrated in FIG. 27G prior to t₀, data ownership information 2710with OSN 6 is determined at t⁻², and where a query 0 is initiated at t⁻¹utilizing OSN 6. Data ownership information 2710 with OSN 6 isillustrated in FIG. 27G. In particular, node 37-1 owns segments ofmemory v, including segments 3, 4, and Y, as physical segments, forexample, based on the storage cluster determining, during execution ofthe corresponding consensus protocol, that these nodes are available asphysical segments stored in memory drive 2440-2 of node 37-1's segmentstorage 2442, based on the failure at to not having yet occurred. Asillustrated in FIG. 27I, node 37-1 generates the segment set for query 0in accordance with OSN 6, where segments 3 and Y are included asphysical segments.

However, due to the failure of memory drive 2440-2, for example, priorto retrieval of segment 3 or segment Y by node 37-1 to execute query 0,the node 37-1 indicates failure in continuing to execute query 0. Thiscan be communicated across the storage cluster and/or the databasesystem to halt other executions by other nodes of query 0 or tootherwise not return a resultant of the query due to the execution ofquery 0 by node 37-1 failing. The time of failure is indicated intimeline 2740 as t_(2.5), but can alternatively be any time after to. Ingeneral, nodes 37 can abort and/or indicate failure of any queries theyexecute that cannot be executed in accordance with the data ownershipinformation assigned to them. In particular, in this example, node 37-1has already determined new data ownership information OSN 7 prior tothis error occurring. However, rather than attempting to continueexecution the query via utilization of the virtual segments indicated inOSN 7, execution of the query is aborted, as utilization of OSN 7mid-query can cause other conflicting ownership problems that couldrender the query incorrect, and/or the correctness of the queryresultant is not guaranteed if the node were to change data ownershipinformation version being utilized for the query after its begunexecuting under a prior version.

In this example, query 1 can correspond to a re-execution of query 0,and thus query 0 can be re-executed as query 1 by the nodes in thestorage cluster based on receiving the updated data ownershipinformation 2710 and based on execution of query 0 previously beingaborted. Query 0 is re-executed as query 1 in accordance with OSN 7.This is acceptable, as all nodes in the storage cluster will re-executequery 0 as query 1 under the same data ownership information, andexecution of query 1 under OSN 7 is maintained by all nodes includingnode 37-1 for the duration of query 1's execution.

As illustrated in FIG. 27J, query 1 is determined to be executed by node37-1 and is tagged to OSN 7. Query 1 is included in the query set withsegments 3 and Y indicated as virtual segments based on the dataownership information of OSN 7. As segments 3 and Y can be recovered viathe recovery scheme in response to being indicated for processing asvirtual segments, in this example, execution of query 1 does not failand its execution is completed at time t₆. Thus, query 0 is ultimatelyexecuted by the storage cluster when it is re-executed as query 1 withthe data ownership information of OSN 7.

In various embodiments, a node of a computing device has at least oneprocessor and memory that stores executable instructions that, whenexecuted by the at least one processor, cause at least one processingmodule of the node to determine first data ownership information viaparticipation in a first execution of a consensus protocol mediated witha plurality of other nodes in a storage cluster that includes the node.The first data ownership information indicates a first ownershipsequence number. The first data ownership information further indicatesthe node's ownership of a first subset of a set of segments, where theset of segments is in a segment group stored by the plurality of nodesin the storage cluster. The executable instructions, when executed bythe at least one processor, further cause the least one processingmodule of the node to determine second data ownership information viaparticipation in a second execution of the consensus protocol mediatedwith the plurality of other nodes in the storage cluster. The seconddata ownership information indicates a second ownership sequence numberthat is different from the first ownership sequence number. The seconddata ownership information further indicates the node's ownership of asecond subset of the set of segments, and where a set difference betweenthe first subset and the second subset is non-null. The at least oneprocessing module of the node receives a first query for execution anddetermines an ownership sequence number tag for the first query thatindicates the value of the first ownership sequence number. The at leastone processing module of the node facilitates execution of the firstquery by utilizing the first subset of the set of segments based ondetermining the ownership sequence number tag of the first queryindicates the value of the first ownership sequence number.

FIG. 27K illustrates a method for execution by a node 37. For example,the node can utilize at least one processing module of the node 37 toexecute operational instructions stored in memory accessible by thenode, where the execution of the operational instructions causes thenode 37 to execute the steps of FIG. 27K. The method of 27K can beperformed by a node 37 in accordance with embodiments of node 37discussed in conjunction with FIGS. 27A-27J, and/or in conjunction withother embodiments of node 37 discussed herein.

Step 2782 includes determining first data ownership information viaparticipation in a first execution of a consensus protocol mediated witha plurality of other nodes in a storage cluster that includes the node.The first data ownership information indicates a first ownershipsequence number. The first data ownership information further indicatesthe node's ownership of a first subset of a set of segments, where theset of segments is in a segment group stored by the plurality of nodesin the storage cluster.

Step 2784 includes determining second data ownership information viaparticipation in a second execution of the consensus protocol mediatedwith the plurality of other nodes in the storage cluster. The seconddata ownership information indicates a second ownership sequence numberthat is different from the first ownership sequence number. The seconddata ownership information further indicates the node's ownership of asecond subset of the set of segments, where a set difference between thefirst subset and the second subset is non-null;

Step 2786 includes receiving or otherwise determining a first query forexecution. Step 2788 includes determining an ownership sequence numbertag for the first query that indicates the value of the first ownershipsequence number. Step 2790 includes facilitating execution of the firstquery by utilizing the first subset of the set of segments based ondetermining the ownership sequence number tag of the first queryindicates the value of the first ownership sequence number.

In various embodiments, the set difference between the first subset andthe second subset is based on a migration of at least one segment in thesegment group and/or a failure of a memory drive that stores at leastone segment in the segment group. In various embodiments, the seconddata ownership information is determined more recently than the firstdata ownership information, and the values of the first ownershipsequence number and the second ownership sequence number indicate thesecond data ownership information is more recent than the first dataownership information. In various embodiments, the first data ownershipinformation indicates that at least one of the first set of the set ofsegments be accessed by the node as a physical segment, and theownership information indicates that at least one of the first set ofthe set of segments be accessed by the node as a virtual segment.

In various embodiments execution of the first query is facilitated bythe node during a temporal period. The temporal period begins prior todetermining the second data ownership information, and the temporalperiod ends after determining the second data ownership information. Invarious embodiments, determining the ownership sequence number tag ofthe first query includes assigning the value of the first ownershipsequence number to the ownership sequence number tag of the first queryin response to receiving the first query based on determining the firstdata ownership information is the most recently determined dataownership information of a plurality of previously determined dataownership information.

In various embodiments, the first query is received and/or determinedfor execution by some or all of the plurality of other nodes in thestorage cluster. Each of this the plurality of other nodes that receivedthe first query can determine the ownership sequence number tag for thefirst query that indicates the value of the first ownership sequencenumber. Each of this plurality of other nodes utilize the first dataownership information to execute the query based on determining theownership sequence number tag for the first query that indicates thevalue of the first ownership sequence number.

In various embodiments, the first query is received in a query request,and determining the ownership sequence number tag for the first query isbased on extracting the ownership sequence number tag for the firstquery from the query request. In various embodiments, another processingmodule of the database system assigns the first ownership sequencenumber to the ownership sequence number tag of the first query, forexample, in response to receiving the first query, based on determiningthe first data ownership information is the most recently determineddata ownership information of a plurality of previously determined dataownership information. This another processing module can assign and/orsend the first query, for example, by generating the query requestindicating the first query and the ownership sequence number tagindicating the first ownership sequence number. The query request can besent by the another processing module to some or all of the plurality ofnodes of in storage cluster that includes the node. The node, as and/orthe other nodes receiving the query request, can determine the ownershipsequence number tag of the first query based on receiving the ownershipsequence number tag assigned to the first query in the query requestreceived from the another processing module.

In various embodiments, each of the plurality of other nodes determinethe first data ownership information via their participation in thefirst execution of the consensus protocol. In various embodiments, thefirst data ownership information indicates a plurality of subsets of theset of segments, where each subset in the plurality of subsets ismutually exclusive and collectively exhaustive with respect to the setof segments. The first data ownership information can indicate each nodein the plurality of other node is mapped to ownership of exactly one ofthe plurality of subsets. Each subset in the plurality of subsets can bemapped to exactly one of: one of the plurality of other nodes or thenode itself.

In various embodiments, the method includes receiving a second queryrequest indicating a second query for execution, where the second queryrequest includes an ownership sequence number tag that indicates thevalue of the second ownership sequence number. The method furtherincludes facilitating execution of the second query by utilizing thesecond subset of the set of segments based on determining the ownershipsequence number tag of the second query indicates the value of thesecond ownership sequence number. In various embodiments, execution ofthe first query is facilitated by the node within a first temporalperiod, and execution of the second query is facilitated by the nodewithin a second temporal period. In various embodiments, an overlapand/or intersection between the first temporal period and the secondtemporal period is non-null.

In various embodiments, facilitating execution of the first query byutilizing the first subset of the set of segments includes utilizingonly segments in the first subset of the set of segments to facilitateexecution of the first query. In various embodiments, segments of theset of segments that are included in the set difference and that arefurther included in the second subset are not utilized by the node infacilitating execution of the first query. In various embodiments,facilitating execution of the first query by utilizing the first subsetof the set of segments further includes utilizing every segment in thefirst subset of the set of segments to facilitate execution of the firstquery. In various embodiments facilitating execution of the first queryby utilizing the first subset of the set of segments further includesselecting a proper subset of the first subset of the set of segmentsbased on identifying only segments of the first subset of the set ofsegments that are required for execution of the first query and byfurther identifying every segment of the first subset of the set ofsegments that are required for execution of the first query every.Facilitating execution of the first query by utilizing the first subsetof the set of segments includes only utilizing this selected propersubset of the first subset of the set of segments to execute the firstquery.

In various embodiments, the method further includes determining at leastone segment in the first subset of the set of segments is unavailable tothe node. The method further includes terminating and/or abortingexecution of the first query based on determining the at least onesegment in the first subset of the set of segments is unavailable to thenode. The method further includes sending a query error messageindicating the termination of execution of the first query, for example,for receipt by other nodes in the storage cluster. In variousembodiments, one other one of the plurality of other nodes facilitatesexecution of the first query utilizing a second subset of the set ofsegments that is distinct from the first subset of the set of segments,and a final query result of the query is not generated based on thequery error message.

In various embodiments, the method includes receiving an error messagegenerated by another node in the storage cluster, for example, where theanother node generates the error message based on the another node beingunable to access at least one segment in the one of the plurality ofsubsets mapped to the another node in the first data ownershipinformation. The method further includes determining to abort executionof the first query and/or to forego transmission of a query resultantgenerated for the first query based on receiving the error message.

In various embodiments the method includes facilitating re-execution ofthe first query by determining an updated ownership sequence number tagfor the first query that indicates the value of the second ownershipsequence number, and by utilizing the second subset of the set ofsegments based on determining the updated ownership sequence number tagfor the first query that indicates the value of the second ownershipsequence number. For example, the method can include determining tofacilitate the re-execution of the first query based on previousfailure, aborting, and/or termination of the first query. The methodincludes determining re-execution of the first query is successful basedon determining all of the segments in the second subset of the set ofsegments are available to the node. In various embodiments, the leastone segment in the first subset of the set of segments is unavailable tothe node during the re-execution of the first query, and there-execution of the first query is successful based on the second subsetnot including the at least one segment.

In various embodiments, the least one segment in the first subset of theset of segments is indicated as at least one corresponding physicalsegment, for example, based on being determined to be available to thenode as a physical segment in the first execution of the consensusprotocol. The at least one segment in the second subset of the set ofsegments is indicated as at least one corresponding virtual segmentbased on the at least one segment in the first subset of the set ofsegments being determined unavailable to the node as the at least onecorresponding physical segment, for example, in the second execution ofthe consensus protocol. Re-execution of the first query can includeperforming a rebuilding function to utilize the at least onecorresponding virtual segment.

In various embodiments, a non-transitory computer readable storagemedium includes at least one memory section that stores operationalinstructions that, when executed by a processing module that includes aprocessor and a memory, cause the processing module to determine firstdata ownership information via participation in a first execution of aconsensus protocol, where the first data ownership information indicatesa first ownership sequence number and further indicates ownership of afirst subset of a set of segments. The set of segments is in a segmentgroup stored by a storage cluster. The operational instructions, whenexecuted by the processing module, further causes the processing moduleto determine second data ownership information via participation in asecond execution of the consensus protocol, where the second dataownership information indicates a second ownership sequence number thatis different from the first ownership sequence number. The second dataownership information further indicates the node's ownership of a secondsubset of the set of segments, where a set difference between the firstsubset and the second subset is non-null. The processing module receivesa first query for execution and determines an ownership sequence numbertag for the first query that indicates the value of the first ownershipsequence number. The processing module facilitates execution of thefirst query by utilizing the first subset of the set of segments basedon determining the ownership sequence number tag of the first queryindicates the value of the first ownership sequence number.

FIGS. 28A-28M illustrate example embodiments where a node 37 is operableto determine one or more levels of a query execution plan 2805 to whichit is assigned to execute incoming queries, and to further determine,for different incoming queries which levels of its assigned set oflevels in which it will participate to execute the incoming queries. Themechanisms utilized by nodes 37 in the database system 10 determiningthis level assignment information as discussed in conjunction with FIGS.28A-28M can be the same as or similar to the mechanisms for determiningdata ownership information 2710 as discussed in conjunction with FIGS.27A-27L. Furthermore, level assignment information can change over timein a same or similar fashion as discussed in conjunction with changes inthe data ownership information 2710, for example, via multiple nodesexecution of a corresponding consensus protocol. The embodimentsillustrated in 28A-28M can be utilized to implement some or all of theplurality of nodes 37 of some or all computing devices 18-1-18-n, forexample, of the of the parallelized data store, retrieve, and/or processsub-system 12, and/or of the parallelized query and results sub-system13. The embodiments of node 37 discussed in conjunction with FIGS.28A-28M can be utilized to implement any other nodes 37 of databasesystem 10 discussed herein. T

FIG. 28A illustrates an example of a query execution plan 2805implemented by the database system 10. The query execution plan caninclude a plurality of levels 2810. In this example, a plurality of Hlevels in a corresponding tree structure of the query execution plan2805 are included. The plurality of levels can include a top, root level2812; a bottom, IO level 2816, and one or more inner levels 2814. Insome embodiments, there is exactly one inner level 2814, resulting in atree of exactly three levels 2810.1, 2810.2, and 2810.3, where level2810.H corresponds to level 2810.3. In such embodiments, level 2810.2 isthe same as level 2810.H-1, and there are no other inner levels2810.3-2810.H-2. Alternatively, any number of multiple inner levels 2814can be implemented to result in a tree with more than three levels.

This illustration of query execution plan 2805 illustrates the flow ofexecution of a given query by utilizing only a proper subset of nodesacross some or all of the levels 2810. In this illustration, nodes 37with a solid outline are nodes involved in executing a given query.Nodes 37 with a dashed outline are other possible nodes that are notinvolved in executing the given query, but could be involved inexecuting other queries in accordance with their level of the queryexecution plan in which they are included, as discussed in furtherdetail herein.

Each of the nodes of IO level 2816 can be operable to, for a givenquery, perform the necessary row reads for gathering corresponding rowsof the query. These row reads can correspond to the segment retrievaldiscussed herein as physical segments or virtual segments to read someor all of the rows of retrieved segments determined to be required forthe given query. Thus, the nodes 37 in level 2816 can include any nodes37 with functionality discussed in conjunction with FIGS. 24A-27L toretrieve segments for query execution; to determine whether to retrievesegments as physical or virtual segments; and/or to determine whichexact set of segments is assigned to the node for retrieval to ensurequeries are executed correctly.

IO level 2816 can include all nodes in a given storage cluster 35 and/orcan include some or all nodes in multiple storage clusters 35, such asall nodes in a subset of the storage clusters 35-1-35-z and/or all nodesin all storage clusters 35-1-35-z. For example, all nodes 37 and/or allcurrently available nodes 37 of the database system 10 can be includedin level 2816. As another example, IO level 2816 can include a propersubset of nodes in the database system, such as some or all nodes thathave access to segment storage 2442 and/or that are included in asegment set 35. In some cases, nodes 37 that are not included in segmentsets, that do not have access to segment storage 2442, and/or that arenot operable to perform row reads are not included at the IO level, butcan be included at one or more inner levels 2814 and/or root level 2812.

The query executions discussed herein by nodes in accordance withexecuting queries at level 2816 can include retrieval of segments;extracting some or all necessary rows from the segments with some or allnecessary columns; and sending these retrieved rows to a node at thenext level 2810.H-1 as the query resultant generated by the node 37. Foreach node 37 at IO level 2816, the set of raw rows retrieved by the node37 can be distinct from rows retrieved from all other nodes, forexample, based on the segment assignment discussed in conjunction withFIGS. 27A-27L, to ensure correct query execution.

Each inner level 2814 can include a subset of nodes 37 in the databasesystem 10. Each level 2814 can include a distinct set of nodes 37 and/orsome or more levels 2814 can include overlapping sets of nodes 37. Thenodes 37 at inner levels are implemented, for each given query, toexecute queries in conjunction with operators for the given query. Forexample, a query plan can be generated for a given incoming query, wherean ordering of execution of its operators is determined, and thisordering is utilized to assign one or more operators to each node in agiven inner level 2814 for execution, for example, as operator data 2416of its queries in query set 2415 of the node. For example, each node ata same inner level can be operable to execute a same set of operatorsfor a given query, in response to being selected to execute the givenquery, upon incoming resultants generated by nodes at a directly lowerlevel to generate its own resultants sent to a next higher level. Incases where there is exactly one inner level, each node selected toexecute a query at a given inner level performs some or all of the givenquery's operators upon the raw rows received as resultants from thenodes at the IO level. In some cases, some operators beyond row readsare also performed by the nodes at the IO level. Each node at a giveninner level 2814 can further perform a gather function to collect,union, and/or aggregate resultants sent from a previous level, forexample, in accordance with one or more corresponding operators of thegiven query.

The root level 2812 can include exactly one node for a given query thatgathers resultants from every node at the top-most inner level 2814. Thenode 37 at root level 2812 can perform additional query operators of thequery and/or can otherwise collect, aggregate, and/or union theresultants from the top-most inner level 2814 to generate the finalresultant of the query, which includes the resulting set of rows and/orone or more aggregated value, in accordance with the query, based onbeing performed on all rows requested by the query. The root level nodecan be selected from a plurality of possible root level nodes, wheredifferent root nodes are selected for different queries. Alternatively,the same root node can be selected for all queries.

As will be discussed in further detail herein, each level can include apossible set of nodes assigned to process queries at the given level.For a given query, a subset of nodes at some or all levels will beselected to process the given query, where different sets nodes areselected to process different queries. In particular, a proper subset ofnodes at each inner level will be selected to process a given query, anddifferent proper subsets are selected for different queries. The nodesat the corresponding lower level can independently which selected nodefor the next level will process their resultant for the given query,where nodes at the corresponding lower level select their parent nodefrom a plurality of possible parent nodes in the corresponding innerlevel to which their resultant is sent for processing. Each selectedparent node thus receives resultants from a plurality of correspondingchild nodes that selected the parent node for processing at the nextlevel.

As depicted in FIG. 28A, resultants are sent by nodes upstream withrespect to the tree structure of the query execution plan as they aregenerated, where the root node generates a final resultant of the query.While not depicted in FIG. 28A, nodes at a same level can share dataand/or send resultants to each other, for example, in accordance withoperators of the query at this same level dictating that data is sentbetween nodes.

In some cases, the IO level 2816 always includes the same set of nodes37, such as a full set of nodes and/or all nodes that are in a storagecluster 35 that stores data required to process incoming queries. Insome cases, the lowest inner level corresponding to level 2810.H-1includes at least one node from the IO level 2816 in the possible set ofnodes. In such cases, while each selected node in level 2810.H-1 isdepicted to process resultants sent from other nodes 37 in FIG. 28A,each selected node in level 2810.H-1 that also operates as a node at theIO level further performs its own row reads in accordance with its queryexecution at the IO level, and gathers the row reads received asresultants from other nodes at the IO level with its own row reads forprocessing via operators of the query. One or more inner levels 2814 canalso include nodes that are not included in IO level 2816, such as nodes37 that do not have access to segment storage 2442 and/or that areotherwise not operable and/or selected to perform row reads for some orall queries.

The node 37 at root level 2812 can be fixed for all queries, where theset of possible nodes at root level 2812 includes only one node thatexecutes all queries at the root level of the query execution plan.Alternatively, the root level 2812 can similarly include a set ofpossible nodes, where one node selected from this set of possible nodesfor each query and where different nodes are selected from the set ofpossible nodes for different queries. In such cases, the nodes at innerlevel 2810.2 determine which of the set of possible root nodes to sendtheir resultant to. In some cases, the single node or set of possiblenodes at root level 2812 is a proper subset of the set of nodes at innerlevel 2810.2, and/or is a proper subset of the set of nodes at the IOlevel 2816. In cases where the root node is included at inner level2810.2, the root node generates its own resultant in accordance withinner level 2810.2, for example, based on multiple resultants receivedfrom nodes at level 2810.3, and gathers its resultant that was generatedin accordance with inner level 2810.2 with other resultants receivedfrom nodes at inner level 2810.2 to ultimately generate the finalresultant in accordance with operating as the root level node.

In some cases where nodes are selected from a set of possible nodes at agiven level for processing a given query, the selected node must havebeen selected for processing this query at each lower level of the queryexecution tree. For example, if a particular node is selected to processa node at a particular inner level, it must have processed the query togenerate resultants at every lower inner level and the IO level. In suchcases, each selected node at a particular level will always use its ownresultant that was generated for processing at the previous, lowerlevel, and will gather this resultant with other resultants receivedfrom other child nodes at the previous, lower level. Alternatively,nodes that have not yet processed a given query can be selected forprocessing at a particular level, where all resultants being gatheredare therefore received from a set of child nodes that do not include theselected node.

The configuration of query execution plan 2805 for a given query can bedetermined in a downstream fashion, for example, where the tree isformed from the root downwards. Nodes at corresponding levels aredetermined from configuration information received from correspondingparent nodes and/or nodes at higher levels, and can each sendconfiguration information to other nodes, such as their own child nodes,at lower levels until the lowest level is reached. This configurationinformation can include assignment of a particular subset of operatorsof the set of query operators that each level and/or each node willperform for the query. The execution of the query is performed upstreamin accordance with the determined configuration, where IO reads areperformed first, and resultants are forwarded upwards until the rootnode ultimately generates the query result.

FIG. 28B illustrates an embodiment where the query execution plan issegregated into a plurality of computing clusters 2835, illustrating asubset of possible sets of nodes from each computing cluster that areselected to process a given query. In this illustration, nodes 37 with asolid outline are again nodes involved in executing the given query.Nodes 37 with a dashed outline are again other nodes that are notinvolved in executing the given query, but could be involved inexecuting other queries in accordance with their level of the queryexecution plan in which they are included.

A computing cluster 2835 can be similar to storage clusters 35 and caninclude a set of possible nodes that can operate in accordance with atleast two levels of the query execution plan. A computing cluster 2835can include some or all nodes of exactly one storage cluster 35. Acomputing cluster 2835 can include some or all nodes of multiple storageclusters 35. For example, a computing cluster 2835 can correspond to a“sub-tree” of query execution plan, corresponding to the possible set ofchild nodes and corresponding possible set of parent nodes each childnode will select a single node from to process their resultants. In thisexample, each computing cluster includes exactly two levels: a lowerlevel corresponding to possible child nodes of the computing cluster andan upper level corresponding to possible parent nodes of the computingcluster. The computing cluster can be implemented as a virtual machinecomputing cluster, for example which each node in the clusterimplemented as a virtual machine processing different queries inaccordance with their selected level.

The set of computing clusters illustrated in FIG. 28B can be utilized toimplement an entire, three level query execution plan 2805 with level2810.1 implemented as root level 2812, with level 2810.2 implemented asthe single inner level 2814, and with level 2810.3 implemented as the IOlevel 2810.3. Alternatively, if the query execution plan 2805 includesmore than three levels, these computing clusters can correspond to asubset of the query execution plan's full set of computing clusters. Inparticular, an additional set of computing clusters can includecorresponding subsets of nodes of level 2810.3 their corresponding upperlevel of possible parent nodes for corresponding possible child nodes ofa subsequently lower level than level 2810.3. Alternatively or inaddition, an additional computing cluster can include all possibleparent nodes of computing cluster 2835 as possible child nodes, as wellas possible parent nodes of one or more additional computing clusters2835-1.2-2835-1.N with upper levels at level 2810.1 and lower levels atlevel 2810.2 as additional possible child nodes. This additionalcomputing cluster could include its own set of possible parent nodes inthe next higher level than level 2810.1. Any number of levels of thequery execution plan can thus be implemented by corresponding computingclusters of the sub-trees.

For each given computing cluster 2835, for a given query, some or allpossible child nodes, corresponding to nodes in the lower level of thecomputing cluster, will be assigned to process the query. The nodes withthe solid outline at the lower level of each computing cluster 2835correspond to the selected subset of possible child nodes executing thegiven query for the corresponding computing cluster 2835. For example,if the lower level of the computing cluster is the IO level 2816 of thequery execution plan, the child nodes generate resultants by performingrow reads. This example is illustrated by illustrated computing clusters2835-2.1-2835-2.G that includes a set of nodes from level 2810.2 aspossible parent nodes and includes a set of nodes from level 2810.3 aspossible child nodes, where level 2810.3 in this example is the IOlevel.

As another example, if the lower level of the computing cluster is aninner level 2814 of the query execution plan, the child nodes receiveresultants as input from child nodes of another, subsequently lower,computing cluster by being selected as the parent node for thesubsequently lower computing cluster for the given query, gather theseresultants, and generate their own resultant. This example isillustrated by illustrated computing cluster 2835-1.1 that includes aset of nodes from level 2810.1 as possible parent nodes and includes aset of nodes from level 2810.2 as possible child nodes. In this example,level 2810.1 can be the root level, as illustrated, or can be an innerlevel that is higher than inner level 2810.2.

As illustrated, for each computer cluster 2835, exactly one node at theupper level receives resultants from nodes at the lower level. Thus, foran execution of a given query by a given computing cluster 2835, everyparticipating node at the lower level is operable to select, for examplewithout global coordination, the same, single node at the upper levelthat will process their resultant as a selected parent node from theplurality of possible parent nodes included in the upper level. Eachparticipating node at the lower level thus sends their resultants tothis same selected parent node. The selected parent node for eachillustrated computing cluster in FIG. 28B for executing the given querycorresponds to the one node in the computing cluster's upper level thathas a solid outline, selected over the other nodes in the computingcluster's upper level with dashed outlines. In some embodiments, if theupper level of computer cluster 2835 is the root level, the same singlenode is selected for every query, where the set of possible parent nodesincludes exactly one node.

Alternatively or in addition, for execution of a given query by a givencomputing cluster 2835, each possible node at the lower level isoperable to determine whether or not it is participating in the givenquery. In some embodiments, all nodes at the lower level that receiveresultants from its own child nodes, for example, in accordance with adifferent computing cluster, is automatically determined to beparticipating at the lower level to ensure these resultants continue tobe processed. In such embodiments, all nodes at the lower level that donot receive resultants from its own child nodes, for example, inaccordance with a different computing cluster selecting a differentparent node, is automatically determined to not participate at the lowerlevel, as it has no resultants as input. In cases where the nodes at thelower level are nodes at the IO level, every node included in orotherwise assigned to the lower can determine to participate at thelower level for any given query. For example, every computing clusterwith its lower level as the IO level, such as computing clusters2835-2.1-2835-2.G in FIG. 28B, can determine that every node at thelower level is responsible for performing row reads, for example, inaccordance with data ownership information 2710.

As discussed previously, it is desirable for nodes 37 to operateindependently without global coordination. Utilizing inter-coordinationbetween only nodes within the same computing cluster can aid in reducingglobal coordination. As illustrated in FIG. 28B, each computing clusterwith the same upper and lower level, such as computing clusters2835-1.1-2835-1.G, can include mutually exclusive sets of nodes aspossible nodes in their respective upper and lower levels. Thus, each ofthese computing clusters 2835-1.1-2835-1.G can independently coordinatethe mechanism for selecting a single parent node to which participatingchild nodes will send their resultants. To further reduce globalcoordination, in some embodiments, no computing clusters haveoverlapping sets of nodes. As a particular example, in embodiments withexactly the three levels as illustrated in FIG. 28B, only computingclusters 2835-2.1-2835-2.G are required, and computing cluster 2835-1.1.is not implemented. In such embodiments, the root level includes exactlyone node that all nodes are predetermined to send resultants to forevery query. In such embodiments, every computing cluster in thedatabase system 10 can be mutually exclusive. In some cases, thedatabase system 10 can implement multiple query execution plans 2805 fordifferent queries, for example, operating on different, distinct sets ofdata stored by the corresponding distinct set of nodes at each queryexecution plan 2805's IO level. Alternatively, the database systemimplements the single query execution plan 2805 for all queries.

Each computing cluster can include the same or different number of totalpossible nodes across each of its levels. A computing cluster caninclude the same or different number of possible nodes for some or allof its levels as other computing clusters that include these samelevels. Each computing cluster can include the same or different numberof levels. For a given query, each selected parent node across differentcomputing clusters at the same level can receive resultants from thesame or different number of child nodes. A same or different number ofchild nodes can be participating for a given query in differentcomputing clusters. Computing clusters that include the lower level asthe IO level can include the same or different number of nodes at the IOlevel. In some cases, all nodes at the IO level and/or all availablenodes at the IO level in every one of these computing clusters thatinclude the lower level as the IO level can be included to implementevery query. In some cases, at least one node at the IO level of atleast one computing cluster will not be selected to perform row readsfor some queries.

As illustrated in FIGS. 28C and 28D, each computing cluster 2835 canhave corresponding level assignment information 2840. The levelassignment information 2840 can be utilized by corresponding nodes inthe computing cluster 2835 to determine which levels of the computingcluster 2835 it is assigned to for participation in some or all queries.In particular, the level assignment information 2840 can indicate acluster-level mapping that indicates assignment of each of a pluralityof subsets of the plurality of levels of the computing cluster 2835 to acorresponding one of the set of nodes. A node assigned to a particularlevel in the level in the level assignment information is included as inthe set of possible nodes for that level, where its participation in agiven query can be determined based on the query itself and/or based onwhether the level is a root level, inner level, or IO level,

As illustrated in FIG. 28C, the level assignment information 2840 caninclude, can be represented as, and/or can otherwise indicate aplurality of T level lists 2844-1-2844-T, corresponding to a pluralityof levels of the computing cluster 2835. For example, if a computingcluster only includes an upper level and a lower level, level list 1 cancorrespond to the level list for the upper level, and level list T cancorrespond to the level list for the lower level, where T is equal totwo. In other embodiments, T can include more than two levels for acorresponding computing cluster than includes nodes in more than twolevels of the query execution plan. Each level lists includes a subsetof nodes in the computing cluster that are assigned to the correspondinglevel as a possible node in the set of possible nodes for the level.

In this example, level list 1 includes a list of i nodes that includesnode 1, node 3, node 4, and node X. Level list 1 has correspondingindices 0-(i−1), where node 1 is at index 0 of the list, node 3 is atindex 1 of the list, node 4 is at index 2 of the list, and node X is atindex i−1 of the list. Level list T includes a list of j nodes thatincludes node 2, node 3, node 4, node 5, and node Y. In this example,level list T does not include node 1. For example, if level list Tcorresponds to the IO level of the query execution plan 2805, level listT can include every node in the computing cluster 2835 and/or everyavailable node in the computing cluster 2835 that has access to segmentstorage 2442 and/or that is included in a corresponding storage cluster35 belonging to the computer cluster 2835. For example, node 1 is notincluded in level list T because it does not include or have access tosegment storage 2442 and/or is not included in any storage clusters 35.In some embodiments, each of a computing clusters level lists 2844 caninclude any number of nodes. For example, i can be greater than j, lessthan j, or equal to j.

The level lists 2844 of level assignment information can indicate, canbe utilized to derive, and/or can be derived from a plurality of nodelevel sets 2845.1-2845.Y. This is illustrated in FIG. 28D, which depictsidentical level assignment information as the example of FIG. 28C in adifferent fashion. As illustrated in FIG. 28D, the level assignmentinformation 2840 can include, can be represented as, and/or canotherwise indicate this set of node level sets 2845.1-2845.Y. Each nodein the computing cluster 2835 has a node level set 2845 that can includeone or more levels to which the node is assigned for the computingcluster 2835 as a possible node, or can indicate the node is assigned tono levels of the computing cluster 2835.

As illustrated in FIGS. 28E-28G, for a given query, each node selectedin a lower level, such as level T, to execute a given query at the lowerlevel, must determine which node at the next, higher level, such aslevel 1, to send their resultant by determining which node will beselected from the set of possible nodes for execution of the query atthis next level. While each node in a computing cluster can determinethe levels to which it is assigned as a possible node based on the levelassignment information 2840, the node needs to further determine whetherit is selected for performing the given query at each of its assignedlevels. Furthermore, if a node selected for participation at a lowerlevel, it needs to determine which node of the set of possible nodes isselected to perform the query at the next, higher level within thecomputing cluster and/or in another computing cluster. As illustrated inFIG. 28E, different queries dictate usage of different parent nodes atlevel 1. Therefore, the nodes at level T do not necessarily haveknowledge of which parent node will be utilized until they havedetermined the corresponding query being executed.

Consider the example where the computing cluster includes exactly twolevels: level T and level 1. For a given query, level T of a computingcluster 2835 includes a selected child node set 2854 of the computingcluster that includes one or more selected child nodes from the set ofpossible child nodes that execute the given query at level T, such asevery node in the possible set of nodes for level T when level T is theIO level. Each of these nodes must determine which of the nodes of thepossible parent node set 24852 of level 1 to send their resultant for agiven query, as the selected node can be the same or different fordifferent queries. For example, possible parent node set 2852 for level1 includes the set of i nodes indicated in level list 1. In some cases,for a given query, every child node must select the same parent nodefrom this set of possible parent nodes, where exactly one node in theset of possible parent nodes is selected to process resultants generatedby every node executing the given query in the selected child node set2854. In this example, assume the same level lists 2844-1 and 2844-Tillustrated in the example level assignment information 2840 of FIGS.28C and 28D.

Consider a particular example where two different queries, query 1 andquery 2, are executed by a plurality of computing clusters including theparticular computing cluster depicted in FIG. 28E with the levelassignment information of FIGS. 28C and 28D. As illustrated in FIG. 28F,the nodes 37 in the computing cluster 2835 determine that node 4 isselected as the selected parent node 2853 for execution of query 1. Inparticular, nodes 2-Y participating in query 1 as nodes in the selectedchild node set 2854, based on having received resultants and/or beingincluded in level list T where level T is the IO level, each determine,for example, independently without communicating amongst themselvesafter query 1 is received for execution, that node 4 is to be selectedas the selected parent node. In response to this determination, eachselected child node, once they have generated their resultants, sendtheir resultants to node 4.

Node 4 itself can determine that is responsible for executing query 1 atlevel 1. If node 4 was included in the selected child node set 2854,node 4 can gather its own resultant that was generated in accordancewith execution of query 1 at level T with the resultants received fromthe other nodes in the selected child node set 2854. In some cases, thenode 4 determines the selected child node set, for example, based on thelevel assignment information 2840 and/or based on additionaldeterminations if the selected child node set is selected as a propersubset of the level list T. This can enable node 4 to determine that ithas received resultants from every node at level T of the computingcluster involved in the query. For example, node 4 determines itrequires resultants from node 2-Y. This ensures that node 4's resultantgenerated upon input based on gathering resultants of the child nodes iscorrect by ensuring the gathered input includes every necessaryresultant generated by each selected child node in the selected childnode set.

As illustrated in FIG. 28G, the nodes 37 in the computing cluster 2835determine that a different node, node 3, is selected as the selectedparent node 2853 for execution of a different query 2. Nodes 2-Yparticipating in query 2 as nodes in the selected child node set 2854each similarly determine that node 3 is to be selected as the selectedparent node 2853 for execution of query 2. In response to thisdetermination, each selected child node, once they have generated theirresultants, send their resultants to node 3. Node 3 similarly candetermine that is responsible for executing query 2 at level 1, and cansimilarly ensure that it has received all necessary resultants as inputto generate its resultant in accordance with level 1.

In these examples, if level 1 is not the root level, the node to whichnode 4 sends its resultant for query 1 generated at level 1, and thenode to which node 3 sends its resultant for query 2 generated at level2, can be the same or different. For example, node 3 and 4 can beincluded in the lowest level list of another computing cluster 2835, andcan similarly determine which parent node in a set of possible parentnodes in a next-lowest level list of the other computing cluster 2835,such as the upper list of two level lists in the other computing cluster2835, to send their respective level 1 resultants. In some cases, iflevel 1 corresponds to the highest inner level and the next level in thequery execution plan is the root level that includes exactly onedesignated root node, node 3 and 4 send their resultants to thisdesignated root node, for example, where this is a predetermineddesignated root node that all nodes at level 1 of the presentedcomputing cluster 2835 will always send their resultants to.

In some embodiments, the determination of which node will be selected asa parent receiving resultants can be dictated by the downstreamconfiguration information, for example, where the parent node of theresulting tree structure is known to a node prior to generating itsresultants. In such cases, determination of a parent can be based on aparent previously determining its child nodes in accordance withforwarding configuration information downstream, where each child nodedetermines the parent as the node from whom this configurationinformation was received. As discussed herein, each node can determineits participation in one or more queries as a selected node in one ormore levels, in accordance with a downstream flow in accordance with thesending configuration information, and/or in accordance with an upstreamflow, for example as resultants are generated and propagated upwards.The level participation determination module 2860 can be implemented ateither or both of these stages for nodes to determine theirparticipation. The level assignment data can dictate consistency of theset of node's participating in a query in both the downstream andupstream processes.

FIG. 28H illustrates embodiments where nodes 37, such as all nodes in acomputing cluster and/or all possible nodes in the query execution plan,implement a level participation determination module 2860 to determinewhether or not it participates at particular levels of its computingcluster for incoming queries and/or to determine which node in a nextlevel, if participating at a given level, to which the resultantgenerated at the given level is to be sent by the node 37. The levelparticipation determination module 2860 can be implemented by a givennode by utilizing at least one processing module of the given node 27and/or of the computing device 18 of the given node.

The level participation determination module 2860 of a given node 37 canbe operable to generate query participation data 2815 indicating whetherthe given node 37 is participating in each query at one or more levelsof the query execution plan, and/or indicating which nodes in one ormore computing clusters 2835 the given node belongs to are selected forparticipation in executing each query at one or more levels of each oneor more computing clusters 2835. The level participation determinationmodule 2860 can utilize a received and/or determined set of queries togenerate the query participation data 2815, for example, where levelparticipation determination module 2860 is generated for each individualquery as it is received and/or determined. The level participationdetermination module 2860 can utilize level assignment information 2840for each of one or more computing clusters to which the given node 37belongs to generate the query participation data 2815 for each query.The level assignment information 2840 can be stored by, accessible by,or otherwise determined by the node 37, for example, based onparticipation in the execution of a corresponding consensus protocolmediated across nodes in the computing cluster 2835 that includes thenode as illustrated in FIG. 28I.

The level participation determination module 2860 can utilize one ormore node selection function definitions 2861 dictating one or more nodeselection functions that are performed by the level participationdetermination module 2860 to generate the query participation data 2815for each query. In this example, the node selection function definitions2861 includes a function definition for a level T parent node selectionfunction utilized for nodes participating in level T to select theirparent node that is selected for level 1. The node selection functiondefinitions 2861 can further include other selection functions, such asa function dictating how the child nodes participating in each query areselected. The node selection function definitions 2861 can be stored by,accessible by, or otherwise determined by the node 37, for example,based on participation in the execution of a corresponding consensusprotocol mediated across nodes in the computing cluster 2835 thatincludes the node as illustrated in FIG. 28I, where the node selectionfunction definitions 2861 are indicated in the level assignmentinformation 2840. Alternatively, the node selection function definitions2861 can fixed and/or constant across multiple versions of levelassignment information 2840, as discussed in further detail inconjunction with FIGS. 28I-28M.

As illustrated, FIG. 28H continues the example of FIGS. 28C-28G byillustrating the utilization of the level participation determinationmodule 2860 of node 3 to determine its participation in a set ofincoming queries query 1, query 2, and query 3. The level participationdetermination module 2860 can utilize, for each incoming query, a querydiscriminator number (QD), for example, where the query discriminatornumber is a unique identifier of the query or other number tagged to thequery and/or deterministic based on the query, where every node in thecomputing cluster 2835 can assign or determine the same QD for eachgiven query.

In particular, the level participation determination module 2860generates query participation data 2815 for a given query as a functionof the query's corresponding QD. This can be accomplished by performingat least one parent node selection function 2862 to determine, for agiven level that the node is determined to participate in, which nodethe resultant is to be sent to. If the node itself is selected as theparent, it can further determine which full set of child nodes areselected for a lower level of the query, indicating the full set ofnodes from which resultants should be received as input. In thisexample, every node in the level list T in computing cluster 2835automatically determines to participate in level T because level T isthe IO level. In this example, all nodes 2-Y automatically participatein level T because level T is the IO level, and because they areincluded in level list T. Therefore, due to automatic participation ofnode 3 in every query at level T, a level T parent node selectionfunction 2862 is performed by node 3.

Alternatively, if a node determined not to participate in level T for agiven query, it may not perform the level T parent node selectionfunction 2862 because it would not generate resultants for the querythat need to be sent to a parent node. However, nodes can still performthis function if included as possible parent nodes to determine whetherthey are responsible for performing the query at level 1. In thisexample, node 1 determines it will not participate at level T for anyquery based on not being included at level list T. However, node 1 canstill perform the level T parent node selection function 2862 onincoming queries to determine whether in will perform a query at level1. For queries where node 1 determines it will participate in level 1,it can receive resultants from all other nodes 2-Y. The node 1 canfurther determine resultants from all nodes 2-Y are required based onall nodes 2-Y being determined to be selected for processing the givenquery, for example, because they will be selected to process everyquery.

A level parent node selection function 2862 can be a function of the QDto select a particular node from the level list from which the parentnode is being selected. In this case, the parent node of level T islevel 1, and thus the full level list 1 of possible parent nodes oflevel T is utilized to select which of the nodes in level list 1 is theparent node, as a function of the QD. In particular, the parent nodeselection function 2862 can include an index selection function 2867that generates an index value as output, as a function of the QD of thegiven query. For each QD of each incoming query, an index value isgenerated by index selection function 2867. In particular, an indexvalue of 2 is generated for query 1 based on query 1 having a QD of 42,an index value of 1 is selected for query 2 based on query 2 having a QDof 101, and an index value of 2 is generated for query 3 based on query3 having a QD of 182.

These indices can be input to an index-to-node mapping 2868 of theparent node selection function 2862. The index-to-node mapping 2868 cansimply utilize the level list from which the parent is being selected,and can select the parent node by selecting the node indexed by theindex value in the list. In this example, node 4 is selected for queries1 and 3 because node 4 is at index 2 of level list 1, as illustrated inFIG. 28C. Node 3 is selected for query 2 because node 3 is at index 1 oflevel list 1, as illustrated in FIG. 28C. Alternatively, anotherdeterministic mapping of index to each node indicated by the levelassignment information can be utilized to select the node as adeterministic function of index.

The index selection function 2867 can implemented to generate a uniformand/or substantially uniform distribution of indices with respect to thelevel list with respect to a plurality of QDs assigned to acorresponding plurality of queries. This can help ensure that over time,nodes selected for participation as the selected parent node 2853 isuniformly distributed and/or substantially uniformly distributed toensure that processing load is distributed relatively evenly acrossdifferent ones of the possible parent node set 2852. The assignment ofQDs with respect to the index selection function 2867 can further beimplemented to help ensure that selection of nodes for sequentiallyreceived queries and/or across the plurality of sequential time slicesis evenly dispersed over time, for example, to ensure that a node is notoverloaded with processing too many queries relative to other possiblenodes in its level list at a given point in time.

In some cases, the index selection function 2867 is further a functionof the length and/or size of the level list from which the parent nodeis being selected, in this case level list 1. For example, possibleoutput of the index selection function 2867 is one of the discrete indexvalues 0-(i−1) of level list 1, based on level list 1 having a length ofi, to ensure the index-to-node mapping only has valid indices as input.As a particular example, the index selection function 2867 produces theresulting index by performing a modulo function on the QD of the givenquery with a modulus value equal to the list length to generate theindex utilized to select the node in the corresponding level list forthe given query. In this case, the modulo function utilizes a modulus ofi because the length of level list 1 is equal to i, where the output ofindex selection function 2867 is therefore always one of the integers0-(i−1). Thus, the index selection function 2867 in this case cangenerate the output index as index=QD % i, where % denotes the modulooperation, and where the outputted value of index is thus equal to theremainder of the Euclidian division of the value of QD by the value ofi.

Furthermore, if incoming queries are assigned a random integer valueand/or sequential integer values with respect to the sequence in whichthey are received and/or determined for execution by the databasesystem, the indices can be selected from the integers 0-(i−1) relativelyuniformly, and furthermore can be substantially evenly spaced out overtime when the incoming queries are assigned to the sequential integervalues and when the queries are executed in a similar order to the orderin which they are received and/or determined for execution.

In this example, suppose the number of nodes in level list 1 of thelevel assignment information is equal to 20 and thus i is equal to 20.The index value of 2 is generated for query 2 because 42% 20=2; theindex value of 1 is generated for query 3 because 101% 20=1; and theindex value of 2 is generated for query 3 because 182% 20=2.

Utilizing the modulo function, or another function of list length andQD, to generate the index enables the same parent node selectionfunction 2862 to be used for any computing cluster with different levellists, as this embodiment of parent node selection function 2862 is afunction of the QD upon the level list itself from which the parent nodeis to be selected. Thus, this same parent node selection function can beutilized by the level participation determination module of any node toselect a parent node at one or more levels, as required. This mechanismof selecting the parent node also ensures that all nodes participatingwill select the same parent node, rather than if they were to implementthe parent node selection function 2862 buy merely selecting a randomnode from the level list or by asynchronously increment an index counterto select nodes sequentially from the list over time. In particular,this mechanism relies only on each node in the computing clusteraccessing the same level assignment information 2840, as will be furtherdiscussed herein.

The selected parent nodes for each query are indicated in the queryparticipation data 2815 generated by the level participationdetermination module 2860. This query participation data 2815 indicatesnode 3's participation in level T for each query 1, 2, and 3. Forexample, the indication that node 3 participates in level T, if level Tis the IO level, is utilized by node 3 to determine to generate segmentsets 2418 for each query 1, 2, and 3 for inclusion in the query set 2415utilized to schedule segment retrieval by the segment scheduler module2410.

The query participation indicates which parent node the resultantgenerated for level T should be set by node 3. For query 2, the parentnode indicates the node itself is the parent node, and that theresultant for level T should be retained by the node itself rather thanbeing transmitted to another node. Furthermore, as a result of theparent node of query 2 being the node itself, level 1 is furtherindicated to indicate the node's participation in level 1 for query 2. Aparent node R is determined as the parent node of level 1, for exampleby performing the same or different parent node selection function 2862as the level 1 parent node selection function 2862 to select node R froma plurality of possible parent nodes at the next highest level in thequery execution plan. Alternatively, node R is known to be the parentnode for all queries, for example, due to being a single root node, andis not selected from a plurality of options via a parent node selectionfunction 2862.

If the node is selected as a parent node of level T, while notillustrated, it can further perform a level parent node selectionfunction 2862 for the next level to determine which node its resultantis to be transmitted to. In some cases, if this next level is outside ofa node's given computing cluster's set of levels, because the node isselected as the parent at the upper-most level of the computing cluster,it can deterministically select a same single node for every query, suchas the designated root node discussed previously, where every node inthe computing cluster, when selected as the parent node at theupper-most level selects this same single node.

In other embodiments, the selected parent node of the upper-most levelof the given computing cluster is a selected one of the selected childnode set 2854 of a different computing cluster's lowest level. A set ofG nodes in a selected child node set 2854 of a particular computingcluster's lowest level, if not the IO level, can include each of theexactly one selected parent nodes of G different computing clusters'upper level. The possible parent nodes of each the G lower computingclusters can thus all be included as possible child nodes in the lowerlevel of the particular computing cluster's lower level. In such cases,rather than independently selecting its own child nodes from thepossible child nodes, the particular computing cluster's determinationof their selected set of G child nodes is already predetermined orotherwise accomplished by each of the G computing cluster's in their owndeterminations of which node of their possible parent node is selected.Thus, selection of child nodes trickles “up” the query execution plan:as lower computing clusters select their parent nodes, this dictates theselection of higher computing clusters' child nodes. The selection of acomputing cluster's selected set of child nodes of its lower level canthere be automatically determined to include all child nodes if itslower level is the IO level, and can automatically include the selectedparent nodes from the lower set of G computing clusters if its lowerlevel is not the IO level. In other embodiments, the selection cantrickle “down” the query execution plan, where higher computing clustersselect their child nodes, which dictates the selection of lowercomputing clusters' selected parent nodes.

For example, if the computing cluster 2835 of the example illustrated inFIGS. 28C-28G is computing cluster 2835-2.1 of FIG. 28B, the selectedparent node of the upper-most level of computing cluster 2835-2 isautomatically included in the selected child node set 2854 of computingcluster 2835-1.1's lowest level. In these embodiments, the possibleparent nodes are included in both computing clusters, and can determinetwo separate level assignment information 2840 for each computingcluster and can further determine the same or different parent nodeselection function 2862 to select a parent node from this lowest levelof computing cluster 2835-1.1 to which the query resultant of node atlevel 1 of computing cluster 2835-2.1 is to be sent.

As a particular example, node 3 is included in two computing clusters: afirst computing cluster, corresponding to the particular computingcluster illustrated in the examples of FIGS. 28C-28H, and a second,other computing cluster. In response to being the selected parent nodeof level T for query 2 in the first computing cluster and thus the beingselected as the node participating in level 1 for query 2 in the firstcomputing cluster, node 3 can determine second level assignmentinformation for the second computing cluster in which node 3 is assignedas a possible node in the lowest level of the other computing cluster.For example, this second level assignment information is accessible bynode 3 due to being included in the second computing cluster. Node Ralso a node of the second computing cluster and is included in a list ofmultiple possible nodes in a next level, such as the upper level of twolevels, of the other computing cluster's level assignment information Inresponse to participating in the lowest level of the another computingcluster for query 2, node 3 performs the same or different parent nodeselection function 2862 to select node R as the parent node, and sendsthe resultant of query 2 generated at the upper level of the firstcomputing cluster to node R. For example, a particular index of thelowest level list of the second level assignment information 2840 isselected from the set of possible indices based on the length of thislowest level list as indicated the second level assignment information2840. For example, the modulo function is performed on the QD byutilizing modulus equal to the length of this lowest level list asindicated the second level assignment information, where node R isindicated in the lowest level list as indicated the second levelassignment information at the resulting index outputted by performingthis modulo function. Node R receives other resultants from other nodesparticipating in the lower level of the second computing cluster,utilized as input to generate its own resultant.

FIG. 28I illustrates an embodiment where the level assignmentinformation 2840 for a computing cluster 2835 is generated via executionof a consensus protocol 2850 by the nodes in computing cluster 2835.Rather than necessitating global coordination and/or single entityresponsible for assignment and sharing of level assignment informationas new versions are generated over time, each new version of the levelassignment information of a particular computing cluster 2835 can begenerated via a consensus protocol, which can be executed by some or allnodes 37 in a computing cluster participating in the consensus protocol,where the shared state mediated via the consensus protocol indicates themost updated level assignment information.

This consensus protocol mediated by the nodes 37 in a computing cluster2835 to generate level assignment information 2840 can be the same as orsimilar to the consensus protocol mediated by the nodes 37 in a storagecluster 35 to generate data ownership information 2710 as discussed inconjunction with FIGS. 27A-27K. In some cases, for example, if thecomputing cluster includes the same set of nodes as the set of nodes ofa storage cluster, the execution of the consensus protocol can beutilized to generate both a new version of the data ownershipinformation as well as a new version of the level assignmentinformation.

For example, as illustrated in FIG. 28I, a plurality of consensusprotocol executions 2850 can be performed via the nodes 37-1-37-X in acomputing cluster 2835 over time to generate a corresponding pluralityof versions of level assignment information 2840. For example, asillustrated by timeline 2841, a first consensus protocol execution2850-1 can be mediated across nodes in the storage cluster duringtimespan t_(1.0)-t_(1.1) to generate a corresponding first version oflevel assignment information 2840-1. For example, the first consensusprotocol execution 2850-1 can be initiated at time t_(1.0) by one ormore nodes in the computing cluster 2835, and the first consensusprotocol execution 2850-1 can be completed, for example, where some orall nodes in the computing cluster have determined and/or can access theresulting level assignment information 2840-1 at t_(1.1). At some timeafter t_(1.1), or perhaps instead at some time before the first thefirst consensus protocol execution 2850-1 is complete but after thefirst consensus protocol execution 2850-1 is initiated, a secondconsensus protocol execution 2850-2 can be mediated across the nodes inthe computing cluster 2835 to generate to generate a correspondingsecond version of level assignment information 2810-2 during timespant_(2.0)-t_(2.1). Similarly, a third consensus protocol execution 2850-3can be mediated across the nodes in the computing cluster 2835 togenerate to generate a corresponding third version of level assignmentinformation 2810-3 during timespan t_(3.0)-t_(3.1), and this process cancontinue over time where consensus protocol executions 2850 areperformed to generate corresponding level assignment information 2840over time. Level assignment information 2810-1, 2810-2, and 2810-3 areeach tagged with a respective compute sequence number (CSN) 2846 withvalues of 1, 2, and 3, respectively, or otherwise indicating theordering of the revision with respect to the other revisions.

As discussed herein, consider the times t_(1.1), t_(2,1), t_(3.1), . . ., t_(i.1) of timeline 2841 as the times where the resultingcorresponding versions of level assignment information 2840-1, 2840-2,2840-3, . . . 2840-i, respectively, are available for utilization by thenodes 37 in the corresponding computing cluster 2835 for query executionas a result of consensus protocol executions 2850-1, 2850-2, 2850-3, . .. , 2850-i being completed across the set of nodes in the computingcluster 2835, where i is any ith iteration of executing the consensusprotocol to generate a corresponding ith version of the level assignmentinformation 2840. The CSN for any ith version of the level assignmentinformation can be tagged with a respective CSN 2846 indicating that theversion is the ith version in the ordering, for example, where the valueof the CSN 2846 is equal to or otherwise indicates the value of i.

As illustrated in FIG. 28I, the consensus protocol can be executed viaconsensus protocol communications generated by nodes 37 and/or receivedand processed by nodes 37. For example, each node can implement a levelassignment consensus module 2830, for example, by utilizing at least oneprocessing module of the node. The level assignment consensus module2830 can be utilized by each corresponding node 37 to generate consensusprotocol communications in accordance with the computing cluster'sexecution of the current consensus protocol for transmission to one ormore other nodes in the computing cluster in accordance with thecomputing cluster's execution of the current consensus protocol, forexample, via system communication resources 14. The level assignmentconsensus module 2830 can be utilized by each corresponding node 37 toreceive and/or process consensus protocol communications, generated byother nodes in the computing cluster 2835 in accordance with thecomputing cluster's execution of the current consensus protocol. Theconsensus protocol can be a leader-mediated consensus protocol.Execution of the consensus protocol can include election or otherdetermination of a leader by one or more nodes, voting by one or morenodes, and/or ultimately arriving at a consensus based on the voting bythe one or more nodes to generate and/or communicate the resulting dataownership information.

One or more nodes can initiate a revision of the level assignmentinformation 2840 by initiating a new execution of the consensusprotocol, for example, in response to determining changes toavailability, storage conditions, processing resources, or anotherchange. Alternatively or in addition, new executions of the consensusprotocol to generate revised level assignment information 2840 can occurat scheduled and/or predetermined times.

Because level assignment information is local only to a particularcomputing cluster 2835, each computing cluster of a small number ofnodes can execute the consensus protocol amongst themselves, rather thanrequiring consensus or other coordination across all nodes in thedatabase system. Each of the computing clusters in a plurality ofcomputing clusters can independently generate their own iterativerevisions of their own level assignment information over time in theirown timeline 2841, where at any given point in time, different computingclusters may have independently generated a different number ofrevisions of their level assignment information. This ensures that theexecution of the consensus protocol remains scalable, while ensuringthat all nodes across different computing clusters 35 has consistentlevel assignment information.

As revised level assignment information is determined by particularnodes over time, most recent versions of the level assignmentinformation 2810 can be implemented to execute incoming queries.However, if the node were to immediately adopt the most recent levelassignment information for segment processing in executing queries inquery set 2415, queries could be processed improperly. In particular, asan individual node executes a query over a span of time, if the nodechanges its level assignment determined for the query based on a morerecent versions of the level assignment information 2840 mid-execution,some resultants needed for execution of the query across all nodes canbe never generated and/or can be sent to parent nodes that are notprocessing the query. Furthermore, multiple nodes can be executing thesame query within slightly different time spans based on their ownsegment scheduler module's initiation of execution of a particularquery. Alternatively or in addition, the most recent level assignmentinformation can be received and/or determined by the different nodes atslightly different times. As global coordination is not utilized and asnodes independently execute queries via the segments they determine toown, a mechanism to ensure all nodes execute each given query with thesame level assignment information is required.

In embodiments where the computing cluster includes at least one node ina storage cluster, a new version of the level assignment informationgenerated by the computing cluster can reflect most recent changes inthe data ownership information generated by the storage cluster, forexample, where nodes assigned to the IO level are based on the mostrecent data ownership information. Similarly, a new version of the dataownership information generated by the storage cluster can reflect mostrecent changes in the level assignment information generated by thecomputing cluster, for example, where segment ownership in the dataownership information is determined is based on the set of nodesassigned to the IO level in the level assignment information.

FIGS. 28J-28L illustrate an example of an embodiment of the presentinvention where nodes in a computing cluster utilize CSNs 2846 tagged toand/or determined for each query to determine which corresponding one ofa plurality of level assignment information versions 2840 generated viathe computing cluster's execution of the consensus protocol over timewill be utilized to determine the corresponding query participation data2815 for each query, for example in the node's query set 2415. This canbe the same as or similar to the tagging of OSNs 2720 to each query todetermine the which corresponding one of a plurality of data ownershipinformation versions 2710 will be utilized for each query, as discussedin conjunction with FIGS. 27A-27K. In particular, the CSN of a query isfixed for the life of the query, where all nodes in the computingcluster execute the query in accordance with level assignmentinformation with a matching CSN, regardless of whether other levelassignment information is generating before or during the query'sexecution.

This improves database systems by ensuring that, despite multipleconcurrently running queries by multiple nodes, a computing cluster'sconfiguration for each query has fixed and consistent level assignmentfor each query. This enables detection of mismatches in clusterconfiguration and ensures that the same set of nodes is considered fordownstream and upstream, and/or ensures that the same set of nodes isconsidered for lateral flow.

FIG. 28J illustrates a particular example of timeline 2841 to illustratethe temporal relation between a series of events occurring at particularpoints in time and/or time spans t₁-t₈. At a point in time t₁, levelassignment information 2840-4 with CSN 4 is generated. For example, theexecution of the consensus protocol can be completed at time t₁ torender the resulting level assignment information. At a later point intime t₅, level assignment information 2840-5 with CSN 5 is generated.

A first query, query 2, can be executed by the computing cluster 2835from time t₂-t₄. Time t₂ can correspond to a time at which query 2 wasreceived and/or at which at least one node initiated a partial executionof query 2. Time t₄ can correspond to a time at which execution of query2 by all nodes in the computing cluster assigned to execute query 2 hascompleted. While execution spans of different nodes in the computingcluster may be different based on their own implementation of theirsegment scheduling module 2410, for the purposes of this example, assumethat the time frame that both particular nodes in the computing clusterexecuted query 2 started between t₁ and t₃ and ended between t₃ and t₅.

A second and third query can similarly be executed by the storagecluster from times t₃-t₇ and times t₆-t₈, respectively. Again, forpurposes of this example, assume that the time frame that particularnodes in the computer cluster executed queries 3 and 4 started and endedsubstantially close to these times relative to other points illustratedin the timeline 2841 of this example. Also note that as illustrated, theexecution of queries 2, 3, and 4 is overlapping, to reflect theconcurrent execution of multiple queries implemented by the storagecluster and to further reflect the concurrent execution of multiplequeries implemented by each node 37 in the storage cluster.

FIG. 28K illustrates the example level assignment information 2840-4 and2840-5 generated as discussed in conjunction with example of FIG. 28Jwith CSNs 2846 of 4 and 5, respectively. In particular, continuing theexample discussed in conjunction with FIGS. 28C-28H, the levelassignment information 2840-4 of this example can be the same levelassignment information 2840 of with FIGS. 28C-28H, as illustrated inFIG. 28K. The level assignment information 2840-5 is different fromlevel assignment information 2840-4. In particular, node 4 is no longerincluded in level list 1 or level list T, for example, based on node 4being removed from the cluster or based on node 4 failing at a timeprior to the time when level assignment information 2840-5 was generatedand/or at a time prior to the time when the corresponding consensusprotocol was initiated. For example, the failure of node 4 was detectedby another node in the computing cluster, and this other node initiatedthe consensus protocol to update the level assignment information tolevel assignment information 2840-5 based on failure of node 4 toexclude node 4 from this new version of the level assignmentinformation. For example, node 4 was detected by a selected parent nodefor level 1 for a query based on never receiving node 4's resultantgenerated in accordance with level T for the query.

Thus in this example, level list T can include fewer nodes, such as jminus 1 nodes, due to the removal of node 4. The lengths of level list Tof level assignment information 2840-4 and 2840-5 denote a correspondingnumber of indices j4 and j5, respectively, to denote that the listlengths can be different for these different level lists 1 2844-4 and2844-5, respectively. Alternatively, level list T may have been changedto add additional nodes to maintain the same number of nodes. Level list1 may include fewer nodes, such as i minus 1 nodes, due to the removalof node 4. The lengths of level list 1 of level assignment information2840-4 and 2840-5 denote a corresponding number of indices i4 and i5,respectively, to denote that the list lengths can be different for thesedifferent level lists 1 2844-4 and 2844-5, respectively. Alternatively,level list 1 may have been changed to add additional nodes to maintainthe same number of nodes. In this example, assume that level list 1 oflevel assignment information.

Continuing with this example, the execution of these queries by node 3in accordance with determined CSNs for these queries is reflected inFIG. 28L. In particular, queries 2 and 3 can be the same queriesillustrated in conjunction with FIG. 28H, and level assignmentinformation 2840-4 can be these same as level assignment information2840 of FIGS. 28C-28H. Thus, queries 2 and 3 have the same queryparticipation data 2815 as discussed in conjunction with FIG. 28H. Inparticular, queries 2 and 3 are tagged with CSN 4, and the levelassignment information 2840-4 is therefore utilized by the levelparticipation determination module 2860 based on level assignmentinformation 2840-4 having a corresponding CSN 2846 of 4.

In particular, the level participation determination module 2860 canimplement a level assignment information selection module 2870 to selectthe particular level assignment information 2840 that will be utilizedfor the query, for example, from a plurality of level assignmentinformation 2840 of historical level assignment information accessibleby, stored by, and/or determined by the node 37. Thus, level assignmentinformation 2840-4 is selected for queries 2 and 3 based on queries 2and 3 having the CSN of 4 and level assignment information 2840-4 havingthe matching CSN of 4. Level assignment information 2840-5 is selectedfor query 4 based on query 4 having the CSN of 5 and level assignmentinformation 2840-5 having the matching CSN of 5.

The CSN tagged for incoming queries can be determined in a same orsimilar fashion as the OSN tagged for incoming queries as discussed inconjunction with FIGS. 27A-27K. For example, queries 2 and 3 are taggedwith CSN 4 because they are received and/or initiated at a time when thelevel assignment information 2840 with CSN 4 is the most recent levelassignment information 2840 generated by the computing cluster, asillustrated by timeline 2841. Query 4 is tagged with CSN 5 because it isreceived and/or initiated at a time when the level assignmentinformation 2840 with CSN 5 is the most recent level assignmentinformation 2840 generated by the computing cluster, as illustrated bytimeline 2841.

As illustrated in FIG. 28L, the level T parent node selection function2862 is performed on each query's QD to select a parent node from levellist 1 from the corresponding level assignment information 2840 selectedby level assignment information selection module 2870. In particular,the parent node for queries 2 and 3 is selected from level list 1 oflevel assignment information 2840-4. In particular, the index isgenerated based on the length i.4 of the level list 1, for example wherethe index is selected from integers 0-19 based on i.4 being equal to 20in the particular example discussed in conjunction with FIG. 28H, andwhere nodes 3 and 4 are selected based on generation of indices of 2 and1, where these indices are applied to the level list 1 of levelassignment information 2840-4 based on level assignment information2840-4 being selected for these queries.

However, the parent node for query 4 is selected from level list 1 oflevel assignment information 2840-5. In particular, the index isgenerated based on the length i.5 of the level list 1, for example wherethe index is selected from integers 0-18 based on i.5 being equal to 19in response to the removal of node 4 decrementing the length of levellist 1 in this version of the level assignment information. Nodes 5 isultimately selected, for example, based on the resulting index beingindex 2, and based on node 5 being included in level list 1 of levelassignment information 2840-5 at index 2 as illustrated in FIG. 28K.Note that, while a same index of 2 is generated for both queries 2 and4, different nodes are selected based on accessing the different levellists 1 of the different level assignment information, in response tothese queries having different tagged CSNs.

For example, the same index selection function 2867 can be utilized togenerate each selected index of 2, 1, and 2, despite utilizing differentlevel assignment information, based on being performed as a function ofthe different lengths i.4 and i.5. In particular, the node selectionfunction definitions 2861 can indicate functions, such as the parentnode selection function 2862, to be utilized for any version of thelevel assignment information. Consider the previously discussedparticular example discussed previously where index values of 1 and 2for queries 2 and 3 are generated by performing the modulo functionutilizing a modulus equal to i.4, thus 20, upon the QDs of 101 and 182for queries 2 and 3 (101% 20=1; 182% 20=2). The index value of 2 isgenerated for query 4 by performing the modulo function utilizing amodulus equal to i.5, thus 19, upon the QD of 287 for query 4 (287%19=2).

The query participation data 2815 can indicate whether or not the nodeis selected to participate in each level, based on utilizing the levelassignment information of the corresponding CSN. The query participationdata can further indicate which parent node a resultant, generated bythe node at a level for which it is determined to participate, is to besent, based on utilizing the level assignment information of thecorresponding CSN.

While not illustrated, all other nodes in the computing cluster tagqueries 2, 3, and 4 with the same CSNs of 4, 4, and 5. All nodesdetermine which levels in which they participate for each queryindependently, in parallel and in isolation, utilizing the levelassignment information of the corresponding CSN and utilizing thecorresponding, parent node selection function(s), whether fixed orindicated in the corresponding level assignment information with thecorresponding CSN.

In particular, all nodes of level list 1 of level assignment information2840-4, including node 4, participate at level T for queries 2 and 3based on determining they are selected to participate based on the levelassignment information 2840-4. All nodes of level list T of levelassignment information 2840-4 send their resultants generated for query2 to node 3 based on all determining that node 3 is the single nodeselected from level list 1 to participate at level 1 for query 2, forexample, by performing the same level T parent node selection function2862 utilizing the QD for query 2 and level list 1 of level assignmentinformation 2840-4. All nodes of level list T of level assignmentinformation 2840-4 send their resultants generated for query 3 to node 4based on all determining that node 4 is the single node selected fromlevel list 1 to participate at level 1 for query 3, for example, byperforming the same level T parent node selection function 2862utilizing the QD for query 3 and level list 1 of level assignmentinformation 2840-4.

Furthermore, all nodes of level list 1 of level assignment information2840-5 participate at level T for query based on based on determiningthey are selected to participate based on the level assignmentinformation 2840-5. Even if failing node 4 again becomes availablebefore or during query 4, it will not participate at level T or level 1based on not being included in level list 1 or level list T of levelassignment information 2840-5. All nodes of level list T of levelassignment information 2840-5 send their resultants generated for query4 to node 5 based on all determining that node 5 is the single nodeselected from level list 1 to participate at level 1 for query 4, forexample, by performing the same level T parent node selection function2862 utilizing the QD for query 4 and level list 1 of level assignmentinformation 2840-5.

FIG. 28M illustrates another example timeline 2841 where query 3 wasdetermined to fail based on node 4 failing during query 3's execution attime t_(4.5). For example, the failure of node 4 is rectified by removalof node 4 from the level lists of revised level assignment information2840-5 as discussed in the example of FIGS. 28J-28L. In particular,query 3 fails due to node 4 being selected to participate in level T forquery 3, but failing to generate and/or send its level T resultant forquery 3. In a similar manner as discussed in conjunction with FIGS.27G-27J, queries can be aborted and/or later re-executed in response tofailing by utilizing updated level assignment information. For example,query 3 is re-executed from t_(7.5)-t₉, this time tagged with CSN 5based on being received and/or initiated after level assignmentinformation 2840-5 was generated, with level assignment information2840-5 being the most recent level assignment information. This time,query 3 is successful, for example, because all nodes determined to beassigned to participate in accordance with level assignment information2840-5 were available and/or generated their resultants at theirassigned levels in accordance with level assignment information 2840-5.

In various embodiments, a node of a computing device has at least oneprocessor and memory that stores executable instructions that, whenexecuted by the at least one processor, cause at least one processingmodule of the node to determine level assignment information indicatingthe node's assignment to a subset of levels of a plurality of levels ofa query execution plan. The plurality of levels can include a lowestlevel such as the IO level, at least one inner level, and a highestlevel such as the root level. The executable instructions, when executedby the at least one processing module of the node, further cause theprocessing module to determine a query for execution and determine aquery discriminator number corresponding to the query. The at least oneprocessing module of the node determines to process the query at one ofthe plurality of levels based on determining the one of the plurality oflevels is included in the subset of levels, and generates queryresultant data by facilitating execution of at least one query operatorof the query in accordance with the one of the plurality of levels. Theat least one processing module of the node selects one of a plurality ofpossible parent nodes in a sequentially higher one of the plurality oflevels based on the query discriminator number and sends the queryresultant data to the one of the plurality of possible parent nodes forprocessing.

In various embodiments, a node of a computing device has at least oneprocessor and memory that stores executable instructions that, whenexecuted by the at least one processor, cause at least one processingmodule of the node to determine first level assignment informationindicating assignment to a first subset of levels of a plurality oflevels of a query execution plan based on a first cluster-level mappinggenerated by participation in a first execution of a consensus protocolmediated via a set of nodes in a computing cluster that includes thenode. The first level assignment information indicates a first computesequence number. The executable instructions, when executed by the atleast one processor, further cause at least one processing module of thenode to determine second level assignment information indicatingassignment to a second subset of levels of the plurality of levels ofthe query execution plan based on a second cluster-level mappinggenerated by participation in a second execution of the consensusprotocol mediated via the set of nodes in the computing cluster. Thesecond level assignment information indicates a second compute sequencenumber that is different from the first compute sequence number.

The at least one processing module of the node determines a query forexecution and determines a compute sequence number tag for the querythat indicates the value of the first compute sequence number. The atleast one processing module of the node determines to utilize the firstlevel assignment information based on determining the compute sequencenumber tag for the query that indicates the value of the first computesequence number. The at least one processing module of the nodedetermines to process the query at one of the plurality of levels basedon determining the one of the plurality of levels is included in thefirst subset of levels and based on determining to utilize the firstlevel assignment information. The at least one processing module of thenode generates query resultant data by facilitating execution of atleast one query operator of the query in accordance with the one of theplurality of levels based on determining to process the query at the oneof the plurality of levels.

FIG. 28N and FIG. 28O illustrate methods for execution by a node 37. Forexample, the node can utilize at least one processing module of the node37 to execute operational instructions stored in memory accessible bythe node, where the execution of the operational instructions causes thenode 37 to execute the steps of FIG. 28N and/or FIG. 28O. The methods ofFIG. 28N and/or FIG. 28O can be performed by a node 37 in accordancewith embodiments of node 37 discussed in conjunction with FIGS. 28A-28M,and/or in conjunction with other embodiments of node 37 discussedherein.

As illustrated in the method of FIG. 28N, step 2872 includes determininglevel assignment information indicating the node's assignment to asubset of levels of a plurality of levels of a query execution plan. Thesubset of levels can be a proper subset of the plurality of levels orcan include all of the plurality of levels. The plurality of levelsincludes a lowest level, at least one inner level, and a highest level.In various embodiments, the plurality of levels includes exactly threelevels. Step 2874 includes receiving and/or otherwise determining aquery for execution. Step 2876 includes determining a querydiscriminator number corresponding to the query. Step 2878 includesdetermining to process the query at one of the plurality of levels basedon determining the one of the plurality of levels is included in thesubset of levels. Step 2880 includes generating query resultant data byfacilitating execution of at least one query operator of the query inaccordance with the one of the plurality of levels based on determiningto process the query at one of the plurality of levels. Step 2882includes selecting one of a plurality of possible parent nodes in asequentially higher one of the plurality of levels based on the querydiscriminator number. Step 2884 includes sending the query resultantdata to the one of the plurality of possible parent nodes forprocessing.

In various embodiments, the level assignment information is determinedbased on a cluster-level mapping generated by execution of a consensusprotocol mediated via a set of nodes in a computing cluster thatincludes the node. The cluster-level mapping indicates assignment ofeach of a plurality of subsets of the plurality of levels to acorresponding one of the set of nodes. In various embodiments,determining the level assignment information includes selecting thecluster-level mapping from a plurality of sequential cluster-levelmappings that has a corresponding compute sequence number that comparesfavorably to a computer sequence number determined for the query. Invarious embodiments, the query execution plan includes assignment of aplurality of nodes in a plurality of computing clusters that includesthe computing cluster. Each of the plurality of computing clustersincludes a subset of the plurality of nodes, and each of the pluralityof computing clusters independently generates corresponding levelassignment information based on executing its own consensus protocolmediated via the corresponding subset of the plurality nodes. In variousembodiments, the cluster-level mapping indicates assignment of the setof nodes to only nodes in a set of two or more consecutive levels of thequery execution plan, where the set of two or more consecutive levels isa proper subset of the plurality of levels.

In various embodiments, the method includes determining the plurality ofpossible parent nodes based on the level assignment information. Invarious embodiments, each of the plurality of possible parent nodes arelisted for assignment to the sequentially higher one of the plurality oflevels in the level assignment information with one of a set ofcorresponding indices. In various embodiments, selecting one of theplurality of possible parent nodes includes performing a deterministicfunction on the query discriminator number to generate a computed index,and further includes selecting the one of the plurality of possibleparent nodes by identifying the one of the plurality of possible parentnodes with a corresponding index in the set of the corresponding indicesthat matches the computed index. In various embodiments, performing thedeterministic function includes performing a modulo operation on thequery discriminator number, where a modulus of the modulo operation isequal to a number of nodes in the plurality of possible parent nodes.

In various embodiments, the method includes receiving a plurality ofqueries, and further includes determining a query discriminator numberfor each of the plurality of queries. For each of the plurality ofqueries, the method includes determining whether to process the eachquery at the one or more of the plurality of levels. When it isdetermined to process the each query at the inner level of the subset oflevels, query resultant data is generated by facilitating partialexecution of the query in accordance with the inner level of the queryexecution plan, and one of the plurality of possible parent nodes in thesequentially higher one of the plurality of levels is selected based onthe query discriminator number of the each query. The query resultantdata of the each query is to the one of the plurality of possible parentnodes for processing. In various embodiments, a distribution ofselection of the plurality of possible parent nodes in the sequentiallyhigher one of the plurality of levels across all of the plurality ofqueries is in accordance with a uniform distribution based on adistribution of the query discriminator numbers of the plurality ofqueries and/or based on the deterministic function utilized to selectthe one of the plurality of parent nodes.

In various embodiments, determining the query discriminator numbercorresponding to the query includes receiving a query request thatincludes the query, and further includes extracting the querydiscriminator number from the query request. In various embodiments,determining the query discriminator number includes performing adeterministic function upon the query, for example, where thedeterministic function is indicated in the level assignment information.

In various embodiments, the plurality of possible parent nodes includesthe node based on the level assignment information indicating assignmentto the sequentially higher one of the plurality of levels. The queryresultant data is processed by the node in accordance with thesequentially higher one of the plurality of levels of the queryexecution plan to generate subsequent query resultant data when the nodeitself is selected from the plurality of possible parent nodes toprocess the query resultant data.

The method can further include receiving a set of query resultant datafrom a corresponding set of nodes, where each of the set of queryresultant data is generated by a corresponding one of the correspondingset of nodes in accordance with the inner level of the query executionplan, and where each of the corresponding set of nodes determines tosend their query resultant data to the node based on selecting the nodefrom the plurality of possible parent nodes based on the querydiscriminator number. Generating the subsequent query resultant dataincludes gathering, performing a union upon, aggregating, and/orotherwise processing the query resultant data generated by the nodeitself with the set of query resultant data received from thecorresponding set of nodes in accordance with the sequentially higherone of the plurality of levels of the query execution plan. This caninclude performing an additional at least one operator of the query uponthe resultant data generated by the node itself and the set of queryresultant data received from the corresponding set of nodes. In variousembodiments, each of the set of nodes determines to send their queryresultant data to the node independently, in isolation and/or withoutcoordination. In various embodiments, each of the set of nodesdetermines to send their query resultant data to the node withoutreceiving a request from the node to send their query resultant data.

In various embodiments, the query includes a plurality of operators. Theplurality of operators includes a first subset of operators thatincludes the at least one operator and a second subset of operators thatincludes the additional at least one operator. Full execution of thequery in accordance with the query execution plan can includeperformance of the first subset of operators by nodes executing at theone of the plurality of levels and can further include performance ofthe second subset of operators by nodes executing at the sequentiallyhigher one of the plurality of levels. In various embodiments, theplurality of operators is segmented into the plurality of subsets ofoperators that includes the first subset and the second subset. Eachsubset of the plurality of subsets is performed by nodes at acorresponding one of the plurality of levels.

In various embodiments, determining to process the query at the one ofthe levels is based on a hierarchical position of the one of the levels.In various embodiments, the query is determined to be processed at theone of the plurality of levels based on determining the one of theplurality of levels is the lowest level of the plurality of levels. Invarious embodiments, the lowest level of the plurality of levelscorresponds to an IO level, and the corresponding subset of theplurality of subsets of operators of the query includes select operatorsand/or read operators. In various embodiments, generating the queryresultant data by facilitating execution of the least one query operatorof the query in accordance lowest level includes facilitating reading ofrows required to execute the query, for example, based on retrieval ofcorresponding physical segments from storage and/or based on recovery ofcorresponding virtual segments via a recovery scheme.

In various embodiments, level assignment information is determined basedon the cluster-level mapping generated by execution of the consensusprotocol mediated via the set of nodes in the computing cluster thatincludes the node. The cluster-level mapping indicates assignment of asubset of the set of nodes to the lowest level, where each node in thesubset of the set of nodes in the computing cluster processes the queryin accordance with the lowest level, and where exactly one node of theset of nodes in the computing cluster processes the query in accordancewith the sequentially higher one of the plurality of levels.

In various embodiments, determining the query is to be processed at theone of the plurality of levels includes determining the one of theplurality of levels is an inner level of the plurality of levels andfurther includes determining, based on the query discriminator number,whether or not to process the query at the one of the plurality oflevels in response to determining the one of the plurality of levels isan inner level. In various embodiments, determining whether or not toprocess the query at the one of the plurality of levels in response todetermining the one of the plurality of levels is an inner level isbased on generating function output by performing a function on thequery discriminator number, where the function output indicates one of aplurality of possible nodes. The node determines to process the query atthe one of the plurality of levels when the one of the plurality ofpossible nodes matches or compares favorably to the node itself. Thenode determines not to process the query at the one of the plurality oflevels when the one of the plurality of possible nodes does not match,corresponds to a different node, and/or compares unfavorably to the nodeitself.

In various embodiments, the node determines to process the query at theone of the plurality of levels based on the one of the plurality ofpossible nodes indicated by the function output compares favorably tothe node itself. In various embodiments, the method includes determininga second query for execution and determining a second querydiscriminator number corresponding to the second query. The methodfurther includes determining whether or not to process the second queryat the one of the plurality of levels in response to determining the oneof the plurality of levels is an inner level. The method furtherincludes generating second function output by performing the function onthe second query discriminator number. The method further includesdetermining not to process the second query at the one of the pluralityof levels based on the second one of the plurality of possible nodesindicated by the second function output comparing unfavorably to thenode itself and/or corresponds to a different node. In variousembodiments, the different node determines to perform the second queryat the one of the plurality of levels by performing the function on thesecond query discriminator number and by determining the correspondingfunction output corresponds to the different node itself.

In various embodiments, a non-transitory computer readable storagemedium includes at least one memory section that stores operationalinstructions that, when executed by a processing module that includes aprocessor and a memory, cause the processing module to determine levelassignment information indicating assignment to a subset of levels of aplurality of levels of a query execution plan, where the plurality oflevels includes a lowest level, at least one inner level, and a highestlevel. The operational instructions, when executed by the processingmodule, further causes the processing module to determine a query forexecution and determine a query discriminator number corresponding tothe query. The processing module determines to process the query at oneof the plurality of levels based on determining the one of the pluralityof levels is included in the subset of levels, and generates queryresultant data by facilitating execution of at least one query operatorof the query in accordance with the one of the plurality of levels. Theprocessing module selects one of a plurality of possible parent nodes ina sequentially higher one of the plurality of levels based on the querydiscriminator number and sends the query resultant data to the one ofthe plurality of possible parent nodes for processing.

In the method illustrated in FIG. 28O, step 2886 includes determiningfirst level assignment information indicating assignment to a firstsubset of levels of a plurality of levels of a query execution planbased on a first cluster-level mapping generated by participation in afirst execution of a consensus protocol mediated via a set of nodes in acomputing cluster that includes the node, where the first levelassignment information indicates a first compute sequence number. Step2888 includes determining second level assignment information indicatingassignment to a second subset of levels of the plurality of levels ofthe query execution plan based on a second cluster-level mappinggenerated by participation in a second execution of the consensusprotocol mediated via the set of nodes in the computing cluster, wherethe second level assignment information indicates a second computesequence number that is different from the first compute sequencenumber. Step 2890 includes receiving and/or otherwise determining aquery for execution. Step 2892 includes determining a compute sequencenumber tag for the first query that indicates the value of the firstcompute sequence number. Step 2894 includes determining to utilize thefirst level assignment information based on determining the computesequence number tag for the first query that indicates the value of thefirst compute sequence number.

Step 2896 includes determining to process the query at one of theplurality of levels based on determining the one of the plurality oflevels is included in the first subset of levels and based ondetermining to utilize the first level assignment information. Invarious embodiments, the one of the plurality of levels is not includedin the second subset of levels of the plurality of levels indicated bythe second level assignment information. Step 2898 includes generatingquery resultant data by facilitating execution of at least one queryoperator of the query in accordance with the one of the plurality oflevels based on determining to process the query at the one of theplurality of levels.

In various embodiments, once the first level assignment information isdetermined to be utilized in step 2894, some of all remaining steps 2896and/or 2898 can be implemented based on the method illustrated in FIG.28N, for example, or where the level assignment information of FIG. 28Nis implemented as the first level assignment information determined tobe utilized in step 2894. Thus, step 2878 can be utilized to implementstep 2896 and/or step 2880 can be utilized to implement step 2898.Furthermore, remaining steps of FIG. 28N can be applied to the method ofFIG. 28O, for example, where steps 2882 and/or 2884 are performed afterthe method of FIG. 28O performs step 2898.

In particular, in various embodiments, the method can further includegenerating query resultant data by facilitating partial execution of thequery in accordance with the one of the plurality of levels, The methodcan include determining a plurality of possible parent nodes in asequentially higher one of the plurality of levels indicated in thefirst cluster-level mapping based on determining the compute sequencenumber tag for the query that indicates the value of the first computesequence number. The method can further include selecting one of theplurality of possible parent nodes in a sequentially higher one of theplurality of levels based on the query, for example, by utilizing aquery discriminator of the query. The method can further include sendingthe query resultant data to the one of the plurality of possible parentnodes for processing.

In various embodiments, a set difference between the first subset oflevels and the second subset of levels is non-null. In variousembodiments, the one of the plurality of levels is not included in thesecond subset of levels. In various embodiments, the set differencebetween the first subset of levels and the second subset of levels isnon-null based on a migration of at least one segment stored by thenode, a failure of a memory drive of the node, and/or a failure ofanother node in the computing cluster.

In various embodiments, execution of the query is facilitated by thenode during a temporal period. The temporal period begins prior todetermining the second level assignment information, and the temporalperiod ends after determining the second level assignment information.In various embodiments, the second level assignment information isdetermined more recently than the first level assignment information,and the values of the first compute sequence number and the secondcompute sequence number indicate the second level assignment informationis more recent than the first level assignment information. In variousembodiments, determining the compute sequence number tag of the queryincludes assigning the value of the first compute sequence number to thecompute sequence number tag of the query in response to receiving thequery and/or based on determining the first level assignment informationis the most recently determined level assignment information of aplurality of previously determined level assignment information.

In various embodiments, the method includes receiving a second queryrequest indicating a second query for execution, where the second queryrequest includes a compute sequence number tag that indicates the valueof the second compute sequence number. The method includes determiningto utilize the second level assignment information for the second querybased on determining the compute sequence number tag for the secondquery indicates the value of the second compute sequence number. Invarious embodiments, the method includes determining not to process thesecond query at the one of the plurality of levels based on determiningthe one of the plurality of levels is not included in the second subsetof levels and based on determining to utilize the second levelassignment information.

In various embodiments, the method includes determining not to processthe query at a second one of the plurality of levels based ondetermining the second one of the plurality of levels is not included inthe first subset of levels and based on determining to utilize the firstlevel assignment information for the query. The method further includesdetermining to process the second query at the second one of theplurality of levels based on determining the second one of the pluralityof levels is included in the second subset of levels and based ondetermining to utilize the second level assignment information for thesecond query. The method further includes generating second queryresultant data by facilitating execution of at least one query operatorof the second query in accordance with the second one of the pluralityof levels based on determining to process the second query at the secondone of the plurality of levels. In various embodiments, execution of thequery is facilitated by the node within a first temporal period.Execution of the second query is facilitated by the node within a secondtemporal period, where an overlap between the first temporal period andthe second temporal period is non-null.

In various embodiments, the query is received in a query request, anddetermining the compute sequence number tag for the query is based onextracting the compute sequence number tag for the query from the queryrequest. In various embodiments, the query is received for execution byeach of the plurality of other nodes in the computing cluster. Each ofthe plurality of other nodes determine the compute sequence number tagfor the query indicating the value of the first compute sequence number.Each of the plurality of other nodes utilize the first level assignmentinformation to execute the query based on determining the computesequence number tag for the query indicates the value of the firstcompute sequence number. In various embodiments, some or all of theplurality of other nodes determine the first level assignmentinformation via their participation in the first execution of theconsensus protocol. In various embodiments, first level assignmentinformation indicates a plurality of subsets of the plurality of levels.The first level assignment information indicates each node in theplurality of other nodes is mapped to assignment to levels of one of theplurality of subsets, where each subset in the plurality of subsets ismapped to one of the plurality of other nodes or the node itself.

In various embodiments, the method includes determining at least oneother node in the computing cluster failed to execute the query inaccordance with one of the plurality of levels assigned to the othernode in the first level assignment data. The method further includesaborting and/or terminating execution of the query based on determiningthe at least one other node in the computing cluster failed to executethe query.

In various embodiments the one of the plurality of levels corresponds toan IO level. The method includes determining a set of segments requiredfor the node to execute the query at the IO level based on dataownership information that indicates the set of segments. In variousembodiments, the data ownership information is determined from aplurality of data ownership information based on an ownership sequencenumber tagged to the query that is different from the compute sequencenumber. The method can include performing row reads via the set ofsegments to execute the query at the IO level, for example, where the atleast one operator indicates the row reads and where the queryresultants indicates a plurality of rows of the set of segments. Invarious embodiments, a first subset of the set of segments are retrievedfrom segment storage by the node based on being indicated in the dataownership information as physical segments, and a second subset of theset of segments are recovered by the node via a recovery scheme based onbeing indicated in the data ownership information as virtual segments.

In various embodiments, the method includes determining at least one ofthe set of segments is unavailable for retrieval as a physical segmentand/or for recovery as a virtual segment. The method includes sending anerror message to other nodes in the computing cluster indicating the atleast one of the set of segments is unavailable. In various embodiments,the other nodes abort and/or terminate execution of the query inresponse to the error message and/or in response to detecting the node'sfailure to generate its query resultant data for the query. In variousembodiments, a final query resultant is not generated via full executionof the query execution plan based on the error message.

In various embodiments, the method includes facilitating re-execution ofthe query by determining an updated compute sequence number tag for thefirst query that indicates the value of the second compute sequencenumber, and by performing in only one or more levels of the secondsubset of the plurality of levels based on determining the updatedcompute sequence number tag for the re-execution of the query thatindicates the value of the second ownership sequence number. In variousembodiments, at least one of the set of segments is unavailable to thenode during the re-execution of the first query. In various embodiments,re-execution of the first query is successful based on the IO level notbeing included in second subset of levels in the second level assignmentdata.

In various embodiments, the re-execution of the first query issuccessful further based on an updated data ownership information beingutilized in re-execution of the first query, where the updated ownershipinformation assigns the at least one of the set of segments to adifferent node in the computing cluster, where the different node in thecomputing cluster is assigned to the IO level in the level assignmentdata, and where the different node re-executes the query in accordancewith the IO level by utilizing the at least one of the set of segmentsbased on the updated ownership information. In various embodiments, theupdated ownership information is generated by the computing cluster inconjunction with the second level assignment data via the execution ofthe second execution of the consensus protocol.

In various embodiments, a non-transitory computer readable storagemedium includes at least one memory section that stores operationalinstructions that, when executed by a processing module that includes aprocessor and a memory, cause the processing module to determine firstlevel assignment information indicating assignment to a first subset oflevels of a plurality of levels of a query execution plan based on afirst cluster-level mapping generated by participation in a firstexecution of a consensus protocol. The first level assignmentinformation indicates a first compute sequence number. The operationalinstructions, when executed by the at least one processor, further causeprocessing module to determine second level assignment informationindicating assignment to a second subset of levels of the plurality oflevels of the query execution plan based on a second cluster-levelmapping generated by participation in a second execution of theconsensus protocol mediated. The second level assignment informationindicates a second compute sequence number that is different from thefirst compute sequence number.

The processing module determines a query for execution and determines acompute sequence number tag for the query that indicates the value ofthe first compute sequence number. The processing module determines toutilize the first level assignment information based on determining thecompute sequence number tag for the query that indicates the value ofthe first compute sequence number. The processing module determines toprocess the query at one of the plurality of levels based on determiningthe one of the plurality of levels is included in the first subset oflevels and based on determining to utilize the first level assignmentinformation. The processing module generates query resultant data byfacilitating execution of at least one query operator of the query inaccordance with the one of the plurality of levels based on determiningto process the query at the one of the plurality of levels.

As may be used herein, the terms “substantially” and “approximately”provides an industry-accepted tolerance for its corresponding termand/or relativity between items. Such an industry-accepted toleranceranges from less than one percent to fifty percent and corresponds to,but is not limited to, component values, integrated circuit processvariations, temperature variations, rise and fall times, and/or thermalnoise. Such relativity between items ranges from a difference of a fewpercent to magnitude differences. As may also be used herein, theterm(s) “configured to”, “operably coupled to”, “coupled to”, and/or“coupling” includes direct coupling between items and/or indirectcoupling between items via an intervening item (e.g., an item includes,but is not limited to, a component, an element, a circuit, and/or amodule) where, for an example of indirect coupling, the intervening itemdoes not modify the information of a signal but may adjust its currentlevel, voltage level, and/or power level. As may further be used herein,inferred coupling (i.e., where one element is coupled to another elementby inference) includes direct and indirect coupling between two items inthe same manner as “coupled to”. As may even further be used herein, theterm “configured to”, “operable to”, “coupled to”, or “operably coupledto” indicates that an item includes one or more of power connections,input(s), output(s), etc., to perform, when activated, one or more itscorresponding functions and may further include inferred coupling to oneor more other items. As may still further be used herein, the term“associated with”, includes direct and/or indirect coupling of separateitems and/or one item being embedded within another item.

As may be used herein, the term “compares favorably”, indicates that acomparison between two or more items, signals, etc., provides a desiredrelationship. For example, when the desired relationship is that signal1 has a greater magnitude than signal 2, a favorable comparison may beachieved when the magnitude of signal 1 is greater than that of signal 2or when the magnitude of signal 2 is less than that of signal 1. As maybe used herein, the term “compares unfavorably”, indicates that acomparison between two or more items, signals, etc., fails to providethe desired relationship.

As may be used herein, one or more claims may include, in a specificform of this generic form, the phrase “at least one of a, b, and c” orof this generic form “at least one of a, b, or c”, with more or lesselements than “a”, “b”, and “c”. In either phrasing, the phrases are tobe interpreted identically. In particular, “at least one of a, b, and c”is equivalent to “at least one of a, b, or c” and shall mean a, b,and/or c. As an example, it means: “a” only, “b” only, “c” only, “a” and“b”, “a” and “c”, “b” and “c”, and/or “a”, “b”, and “c”.

As may also be used herein, the terms “processing module”, “processingcircuit”, “processor”, and/or “processing unit” may be a singleprocessing device or a plurality of processing devices. Such aprocessing device may be a microprocessor, micro-controller, digitalsignal processor, microcomputer, central processing unit, fieldprogrammable gate array, programmable logic device, state machine, logiccircuitry, analog circuitry, digital circuitry, and/or any device thatmanipulates signals (analog and/or digital) based on hard coding of thecircuitry and/or operational instructions. The processing module,module, processing circuit, and/or processing unit may be, or furtherinclude, memory and/or an integrated memory element, which may be asingle memory device, a plurality of memory devices, and/or embeddedcircuitry of another processing module, module, processing circuit,and/or processing unit. Such a memory device may be a read-only memory,random access memory, volatile memory, non-volatile memory, staticmemory, dynamic memory, flash memory, cache memory, and/or any devicethat stores digital information. Note that if the processing module,module, processing circuit, and/or processing unit includes more thanone processing device, the processing devices may be centrally located(e.g., directly coupled together via a wired and/or wireless busstructure) or may be distributedly located (e.g., cloud computing viaindirect coupling via a local area network and/or a wide area network).Further note that if the processing module, module, processing circuit,and/or processing unit implements one or more of its functions via astate machine, analog circuitry, digital circuitry, and/or logiccircuitry, the memory and/or memory element storing the correspondingoperational instructions may be embedded within, or external to, thecircuitry comprising the state machine, analog circuitry, digitalcircuitry, and/or logic circuitry. Still further note that, the memoryelement may store, and the processing module, module, processingcircuit, and/or processing unit executes, hard coded and/or operationalinstructions corresponding to at least some of the steps and/orfunctions illustrated in one or more of the Figures. Such a memorydevice or memory element can be included in an article of manufacture.

One or more embodiments have been described above with the aid of methodsteps illustrating the performance of specified functions andrelationships thereof. The boundaries and sequence of these functionalbuilding blocks and method steps have been arbitrarily defined hereinfor convenience of description. Alternate boundaries and sequences canbe defined so long as the specified functions and relationships areappropriately performed. Any such alternate boundaries or sequences arethus within the scope and spirit of the claims. Further, the boundariesof these functional building blocks have been arbitrarily defined forconvenience of description. Alternate boundaries could be defined aslong as the certain significant functions are appropriately performed.Similarly, flow diagram blocks may also have been arbitrarily definedherein to illustrate certain significant functionality.

To the extent used, the flow diagram block boundaries and sequence couldhave been defined otherwise and still perform the certain significantfunctionality. Such alternate definitions of both functional buildingblocks and flow diagram blocks and sequences are thus within the scopeand spirit of the claims. One of average skill in the art will alsorecognize that the functional building blocks, and other illustrativeblocks, modules and components herein, can be implemented as illustratedor by discrete components, application specific integrated circuits,processors executing appropriate software and the like or anycombination thereof.

In addition, a flow diagram may include a “start” and/or “continue”indication. The “start” and “continue” indications reflect that thesteps presented can optionally be incorporated in or otherwise used inconjunction with other routines. In this context, “start” indicates thebeginning of the first step presented and may be preceded by otheractivities not specifically shown. Further, the “continue” indicationreflects that the steps presented may be performed multiple times and/ormay be succeeded by other activities not specifically shown. Further,while a flow diagram indicates a particular ordering of steps, otherorderings are likewise possible provided that the principles ofcausality are maintained.

The one or more embodiments are used herein to illustrate one or moreaspects, one or more features, one or more concepts, and/or one or moreexamples. A physical embodiment of an apparatus, an article ofmanufacture, a machine, and/or of a process may include one or more ofthe aspects, features, concepts, examples, etc. described with referenceto one or more of the embodiments discussed herein. Further, from figureto figure, the embodiments may incorporate the same or similarly namedfunctions, steps, modules, etc. that may use the same or differentreference numbers and, as such, the functions, steps, modules, etc. maybe the same or similar functions, steps, modules, etc. or differentones.

Unless specifically stated to the contra, signals to, from, and/orbetween elements in a figure of any of the figures presented herein maybe analog or digital, continuous time or discrete time, and single-endedor differential. For instance, if a signal path is shown as asingle-ended path, it also represents a differential signal path.Similarly, if a signal path is shown as a differential path, it alsorepresents a single-ended signal path. While one or more particulararchitectures are described herein, other architectures can likewise beimplemented that use one or more data buses not expressly shown, directconnectivity between elements, and/or indirect coupling between otherelements as recognized by one of average skill in the art.

The term “module” is used in the description of one or more of theembodiments. A module implements one or more functions via a device suchas a processor or other processing device or other hardware that mayinclude or operate in association with a memory that stores operationalinstructions. A module may operate independently and/or in conjunctionwith software and/or firmware. As also used herein, a module may containone or more sub-modules, each of which may be one or more modules.

As may further be used herein, a computer readable memory includes oneor more memory elements. A memory element may be a separate memorydevice, multiple memory devices, a set of memory locations within amemory device or a memory section. Such a memory device may be aread-only memory, random access memory, volatile memory, non-volatilememory, static memory, dynamic memory, flash memory, cache memory,and/or any device that stores digital information. The memory device maybe in a form a solid-state memory, a hard drive memory, cloud memory,thumb drive, server memory, computing device memory, and/or otherphysical medium for storing digital information.

While particular combinations of various functions and features of theone or more embodiments have been expressly described herein, othercombinations of these features and functions are likewise possible. Thepresent disclosure is not limited by the particular examples disclosedherein and expressly incorporates these other combinations.

What is claimed is:
 1. A method for execution by a processing system ofa node, comprising: determining, by at least one processor of theprocessing system, first level assignment information indicatingassignment to a first subset of levels of a plurality of levels of acomputing cluster having a plurality of nodes, wherein the first levelassignment information indicates a first compute sequence number;determining, by the at least one processor of the processing system,second level assignment information indicating assignment to a secondsubset of levels of the plurality of levels, wherein the second levelassignment information indicates a second compute sequence number thatis different from the first compute sequence number, wherein the secondlevel assignment information is determined more recently than the firstlevel assignment information; response to receiving a first queryrequest indicating a query for execution: identifying, by the at leastone processor of the processing system, a compute sequence number tagfor the query that indicates a value of the first compute sequencenumber; utilizing, by the at least one processor of the processingsystem, the first level assignment information based on the computesequence number tag for the query that indicates the value of the firstcompute sequence number; processing, by the at least one processor ofthe processing system, the query at one of the plurality of levels basedon determining the one of the plurality of levels is included in thefirst subset of levels and based on the utilized first level assignmentinformation; generating, by the at least one processor of the processingsystem, query resultant data during a temporal period by performingexecution of at least one query operator of the query in accordance withthe one of the plurality of levels based on to the processing of thequery at the one of the plurality of levels, wherein the temporal periodbegins prior to the determining the second level assignment information,wherein the query resultant data is based on data accessed via a segmentstorage; determine a plurality of possible parent nodes in asequentially higher one of the plurality of levels indicated in a firstcluster-level mapping of the first level assignment information based ondetermining the compute sequence number tag for the query; and sending,based on the determining of the plurality of possible parent nodes in asequentially higher one of the plurality of levels, the query resultantdata to one of the plurality of possible parent nodes for processing. 2.The method of claim 1, wherein the one of the plurality of levels is notincluded in the second subset of levels of the plurality of levelsindicated by the second level assignment information.
 3. The method ofclaim 1, further comprising: selecting the one of the plurality ofpossible parent nodes in the sequentially higher one of the plurality oflevels based on the query.
 4. The method of claim 1, wherein thetemporal period ends after determining the second level assignmentinformation.
 5. The method of claim 1, wherein the value of the firstcompute sequence number is assigned to the compute sequence number tagof the query based on determining the first level assignment informationis the most recently determined level assignment information of aplurality of previously determined level assignment information when thequery is received for execution.
 6. The method of claim 1, wherein a setdifference between the first subset of levels and the second subset oflevels is non-null, and wherein the one of the plurality of levels isnot included in the second subset of levels.
 7. The method of claim 6,wherein the set difference between the first subset of levels and thesecond subset of levels is non-null based on at least one of: amigration of at least one segment stored by the node, a failure of amemory drive of the node; or a failure of another node in computingcluster that includes the node.
 8. The method of claim 1, furthercomprising: receiving a second query request indicating a second queryfor execution, wherein the second query request includes the computesequence number tag that indicates value of the first compute sequencenumber; utilizing the first level assignment information based ondetermining the compute sequence number tag for the second queryindicates the value of the first compute sequence number; processing thesecond query at the one of the plurality of levels based on determiningthe one of the plurality of levels is included in the first subset oflevels based on utilizing the first level assignment information; andgenerating second query resultant data during another temporal period byperforming execution of at least one corresponding query operator of thesecond query in accordance with the one of the plurality of levels basedon determining to process the second query at the one of the pluralityof levels, wherein the another temporal period begins after thedetermining the second level assignment information.
 9. The method ofclaim 1, further comprising: receiving a second query request indicatinga second query for execution, wherein the second query request includesa second compute sequence number tag that indicates value of the secondcompute sequence number; and determining to utilize the second levelassignment information based on determining the second compute sequencenumber tag for the second query indicates the value of the secondcompute sequence number; and determining to process the second query ata second one of the plurality of levels based on determining the secondone of the plurality of levels is included in the second subset oflevels and based on determining to utilize the second level assignmentinformation for the second query; and generating second query resultantdata by performing execution of at least one query operator of thesecond query in accordance with the second one of the plurality oflevels based on determining to process the second query at the secondone of the plurality of levels.
 10. The method of claim 9, wherein thesecond one of the plurality of levels is distinct from the one of theplurality of levels, wherein the second query is not processed at theone of the plurality of levels based on determining the one of theplurality of levels is not included in the second subset of levels andbased on determining to utilize the second level assignment information;and wherein the query is not processed at the second one of theplurality of levels based on determining the second one of the pluralityof levels is not included in the first subset of levels and based ondetermining to utilize the first level assignment information for thequery.
 11. The method of claim 9, wherein the second query resultantdata is generated by the node within a second temporal period andwherein an overlap between the first temporal period and the secondtemporal period is non-null.
 12. The method of claim 1, wherein thequery is received in a query request, and wherein determining thecompute sequence number tag for the query is based on extracting thecompute sequence number tag for the query from the query request. 13.The method of claim 1, wherein the first level assignment information isdetermined based on the first cluster-level mapping generated byparticipation in a first execution of a consensus protocol mediated viaa set of nodes in the computing cluster that includes the node during afirst temporal period, and wherein the second level assignmentinformation is determined based on a second cluster-level mappinggenerated by participation in a second execution of the consensusprotocol mediated via the set of nodes in the computing cluster during asecond temporal period after the first temporal period.
 14. The methodof claim 13, wherein the query is received for execution by each node ofa plurality of other nodes in the computing cluster, wherein each nodeof the plurality of other nodes determine the compute sequence numbertag for the query that indicates the value of the first compute sequencenumber, and wherein each node of the plurality of other nodes utilizethe first level assignment information to execute the query based ondetermining the compute sequence number tag for the query that indicatesthe value of the first compute sequence number.
 15. The method of claim14, wherein each node of the plurality of other nodes determine thefirst level assignment information via participation of the other nodesin the first execution of the consensus protocol.
 16. The method ofclaim 1, wherein the first level assignment information indicates aplurality of subsets of the plurality of levels, wherein the first levelassignment information indicates each node in a plurality of other nodesis mapped to assignment of one of the plurality of subsets, and whereineach subset in the plurality of subsets is mapped to one of: one of theplurality of other nodes or the node.
 17. The method of claim 1, whereinvalues of the first compute sequence number and the second computesequence number indicate the second level assignment information is morerecent than the first level assignment information.
 18. A node of acomputing device comprising: at least one processor; and memory thatstores executable instructions that, when executed by the at least oneprocessor, cause at least one processing module of the node to:determine first level assignment information indicating assignment to afirst subset of levels of a plurality of levels of a computing clusterhaving a plurality of nodes, wherein the first level assignmentinformation indicates a first compute sequence number; determine secondlevel assignment information indicating assignment to a second subset oflevels of the plurality of levels, wherein the second level assignmentinformation indicates a second compute sequence number that is differentfrom the first compute sequence number, wherein the second levelassignment information is determined more recently than the first levelassignment information; response to receiving a first query requestindicating a query for execution: identify a compute sequence number tagfor the query that indicates a value of the first compute sequencenumber; utilize the first level assignment information based on thecompute sequence number tag for the query that indicates the value ofthe first compute sequence number; process the query at one of theplurality of levels based on determining the one of the plurality oflevels is included in the first subset of levels and based on theutilized first level assignment information; generate query resultantdata during a temporal period by performing execution of at least onequery operator of the query in accordance with the one of the pluralityof levels based on the processing of the query at the one of theplurality of levels, wherein the temporal period begins prior to thedetermining the second level assignment information, wherein the queryresultant data is based on data accessed via a segment storage;determine a plurality of possible parent nodes in a sequentially higherone of the plurality of levels indicated in a first cluster-levelmapping of the first level assignment information based on determiningthe compute sequence number tag for the query; and send, based on thedetermining of the plurality of possible parent nodes in thesequentially higher one of the plurality of levels, the query resultantdata to one of the plurality of possible parent nodes for processing.19. The node of claim 18, wherein the one of the plurality of levels isnot included in the second subset of levels of the plurality of levelsindicated by the second level assignment information.
 20. Anon-transitory computer readable storage medium comprises: at least onememory section that stores operational instructions that, when executedby a processing module that includes a processor and a memory, causesthe processing module to: determine first level assignment informationindicating assignment to a first subset of levels of a plurality oflevels of a computing cluster having a plurality of nodes, wherein thefirst level assignment information indicates a first compute sequencenumber; determine second level assignment information indicatingassignment to a second subset of levels of the plurality of levels,wherein the second level assignment information indicates a secondcompute sequence number that is different from the first computesequence number, wherein the second level assignment information isdetermined more recently than the first level assignment information;response to receiving a first query request indicating a query forexecution: identify a compute sequence number tag for the query thatindicates a value of the first compute sequence number; utilize thefirst level assignment information based on the compute sequence numbertag for the query that indicates the value of the first compute sequencenumber; process the query at one of the plurality of levels based ondetermining the one of the plurality of levels is included in the firstsubset of levels and based on the utilized first level assignmentinformation; generate query resultant data during a temporal period byperforming execution of at least one query operator of the query inaccordance with the one of the plurality of levels based on theprocessing of the query at the one of the plurality of levels, whereinthe temporal period begins prior to the determining the second levelassignment information, wherein the query resultant data is based ondata accessed via a segment storage; determine a plurality of possibleparent nodes in a sequentially higher one of the plurality of levelsindicated in a first cluster-level mapping of the first level assignmentinformation based on determining the compute sequence number tag for thequery; and send, based on the determining of the plurality of possibleparent nodes in the sequentially higher one of the plurality of levels,the query resultant data to one of the plurality of possible parentnodes for processing.